We are seeking a highly experienced AWS Data Lead / Lead Consultant with strong expertise in designing and implementing scalable cloud-based data engineering solutions. The ideal candidate will lead end-to-end ETL/ELT pipeline development using AWS services and modern data engineering tools, ensuring high performance, reliability, and data quality across enterprise data platforms.
Must-Have Skills
- AWS Glue (Strong hands-on experience)
- PySpark
- dbt (Data Build Tool)
- Apache Airflow
Detailed Responsibilities
- Design, develop, and optimize end-to-end data pipelines using AWS Glue with PySpark.
- Build and manage scalable ETL/ELT workflows and data integration solutions across cloud platforms.
- Develop orchestration workflows using Apache Airflow for scheduling and monitoring data pipelines.
- Design event-driven and scalable architectures using AWS services such as Lambda, EventBridge, S3, RDS (Aurora/PostgreSQL), and Kinesis/MSK.
- Work with relational and cloud-native databases including Snowflake, PostgreSQL, Aurora, and MS SQL Server.
- Implement robust data modeling practices and write optimized SQL queries for analytics and reporting.
- Build and maintain CI/CD pipelines and implement Infrastructure as Code (Terraform or similar tools).
- Ensure high data quality, governance, and performance optimization across all data workflows.
- Collaborate with cross-functional teams and stakeholders to deliver business-critical data solutions.
- Provide technical leadership and mentorship to engineering teams.
Top 3 Key Responsibilities
- Lead the design and development of scalable AWS-based ETL/ELT data pipelines using Glue, PySpark, and Airflow.
- Architect and implement cloud data solutions leveraging AWS services and modern data engineering practices.
- Drive CI/CD, automation, and best practices for reliable, high-quality data platform delivery.