- Design, develop, and maintain scalable ETL pipelines (Vertex AI pipeline) for data integration and transformation and dataflow jobs.
- Build and optimize data models, databases, and data warehouses to ensure efficient data storage and retrieval.
- Collaborate with data analysts, data scientists, and other stakeholders to understand data needs and deliver solutions.
- Ensure data quality, reliability, and security by implementing appropriate validation and governance processes.
- Work with cloud platforms (Google Cloud) to manage and deploy data infrastructure.
- Monitor and troubleshoot performance issues in data pipelines and systems.
- Develop and maintain documentation for data systems, processes, and workflows.
Required Skills and Qualifications:
- Bachelor’s or Master’s degree in Computer Science, Data Science, Engineering, or a related field.
- Proven experience as a Data Engineer or in a similar role.
- Expertise in SQL and database management systems (e.g., MySQL, PostgreSQL, Snowflake).
- Experience with big data technologies (e.g., Hadoop, Spark) and data pipeline tools (e.g., Apache Airflow, Kafka).
- Familiarity with cloud platforms and services (e.g., Google BigQuery).