Job Summary
We are seeking an experienced Hadoop Developer with strong expertise in Hadoop and Kafka ecosystems. The role involves designing, developing, and optimizing big data solutions using Hadoop, Spark, and Kafka within a Cloudera (CDP) environment, primarily in the banking domain.
Required Skills & Experience
- Minimum 8+ years of experience in Hadoop-based development.
- Strong expertise in Hadoop ecosystem including HDFS, MapReduce, and data processing frameworks.
- Hands-on experience with Apache Kafka (brokers, producers, consumers).
- Strong knowledge of Kafka concepts: topics, partitions, offsets, and cluster configuration.
- Experience in Kafka configuration, monitoring, and deployments.
- Strong experience in Apache Spark with Scala or Java.
- Proficiency in performance tuning of big data applications.
- Strong Shell Scripting skills.
- Experience working on Cloudera Data Platform (CDP).
Additional Skills
- Experience with Agile tools such as Jira and Confluence.
- Knowledge of CI/CD pipelines and deployment processes.
- Experience with Git/Bitbucket version control systems.
- Understanding of metadata-driven ETL frameworks.
- Banking domain experience preferred.
Key Responsibilities
- Develop and maintain scalable big data applications using Hadoop, Spark, and Kafka.
- Manage Kafka cluster operations including configuration, monitoring, and troubleshooting.
- Optimize data pipelines for performance and reliability.
- Work on CI/CD deployments and support release activities.
- Collaborate in Agile teams to deliver data engineering solutions.