Key Responsibilities

• Lead the design, implementation, and optimization of PySpark-based solutions for big data processing.
• Collaborate with cross-functional teams to develop efficient ETL pipelines and data models.
• Manage and optimize SQL and NoSQL databases (DB2, PostgreSQL, Snowflake, etc.).
• Implement and manage workflows using Apache Airflow.
• Leverage AWS cloud platforms to build scalable and robust data infrastructure.
• Oversee DevOps processes, including CI/CD pipelines, and containerized environments (Docker, Kubernetes).
• Mentor and guide team members, ensuring high performance and efficient problem-solving.

Requirements

• 10+ years of experience in big data technologies, with a strong focus on PySpark and Apache Spark.
• Extensive experience with Python and database management (SQL, NoSQL).
• In-depth knowledge of ETL processes, data modeling, and data pipeline architectures.
• Hands-on experience with AWS data services (S3, EMR, Redshift, etc.).
• Proficiency with workflow schedulers such as Airflow.
• Familiarity with DevOps practices and tools (Docker, Kubernetes, Jenkins, etc.).
• Strong leadership skills and experience in managing teams of engineers.

Posted By