Key and Responsibilities

• Design, develop, and maintain scalable data pipelines and cloud-based solutions using AWS services such as EMR, S3, Redshift, Glue, Lambda, and DynamoDB.
• Build and manage data streams and batch processing systems using PySpark and Kafka.
• Optimize data storage and querying through partitioning and Parquet file management.
• Write and optimize complex SQL queries and work with databases like Redshift and Snowflake.
• Implement and manage CI/CD pipelines, containerization (Docker), and infrastructure-as-code (Terraform).
• Lead and mentor a team of engineers, ensuring high-quality technical implementations.
• Utilize Redshift Spectrum and Athena to query large datasets.
• Ensure security and compliance by following best practices, including managing IAM roles and data governance.
• Work on data architecture, including data lake, data warehouse, and data lakehouse concepts.
• Maintain high standards of data quality and implement governance frameworks.

Required Skills/Qualification

• 7-10 years of experience in software engineering with a strong focus on data engineering and cloud technologies.
• Extensive hands-on experience with AWS services and ETL concepts.
• Proven leadership and team mentoring capabilities.
• Strong knowledge of security, compliance, and governance for cloud data solutions.

Posted By