Responsibilities

• Develop efficient ETL pipelines using Apache Spark, Python, Kafka, and Hive on Cloudera Data Platform.
• Analyze requirements and translate them into technical designs and estimates.
• Collaborate with data analysts and business users to understand data needs and create source-to-target mappings.
• Design, build, and optimize large-scale data pipelines ensuring SLA compliance.
• Apply performance tuning techniques to Spark jobs and lead optimization efforts.
• Develop data ingestion and governance frameworks.
• Manage production deployments, troubleshoot issues, and handle change requests.
• Lead data migration and historical data rebuild projects.
• Perform code reviews and ensure best practices in development.
• Work independently and with global teams to ensure timely project delivery.

Requirements

• Strong experience with PySpark, Python, Hive, and Hadoop ecosystem (Cloudera preferred).
• Proficient in writing optimized Spark and MapReduce jobs.
• Hands-on experience with Linux/Unix environments.
• Knowledge of Impala and Kafka is a plus.
• Experience with large-scale data pipelines and performance tuning.
• Ability to translate business requirements into technical solutions.
• Good communication skills and ability to work with cross-functional teams.
• Familiarity with cloud technologies is desirable.

Posted By