Roles and Responsibilities

• Design, implement, and maintain robust Kafka-based data pipelines, ensuring high throughput, low latency, and fault tolerance.
• Troubleshoot and resolve Kafka messaging issues, including consumer lag, broker failures, and message delivery problems.
• Optimize Kafka configurations for performance and scalability across development, QA, UAT, and production environments.
• Monitor Kafka clusters using metrics and logs, proactively identifying performance bottlenecks and resolving issues.
• Lead the development of Kafka Streams pipelines and KSQL queries for real-time data processing.
• Administer and configure distributed Kafka clusters, troubleshooting network, connectivity, and configuration issues.
• Implement best practices for Kafka cluster health, backup, and disaster recovery (HA/DR).
• Work closely with cross-functional teams to integrate Kafka with other systems, ensuring seamless data flow.
• Provide technical direction and guidance to other engineers working on Kafka projects.
• Implement and maintain Kafka-based solutions using Agile methodology, ensuring timely delivery and quality.
• Automate processes wherever possible, adhering to the "automate-first" philosophy.
• Lead the troubleshooting and resolution of Kafka-related issues, including performance problems and message delivery failures.
• Develop subsystems and microservices, ensuring optimal integration with Kafka.
• Ensure Kafka security configurations for authentication, authorization, and encryption are correctly implemented.

Required Skills

• In-depth knowledge of Kafka brokers, partitions, topics, consumer groups, and replication mechanisms.
• Strong experience in Java or Scala; knowledge of Python is a plus.
• Ability to design scalable Kafka-based solutions considering throughput, fault tolerance, and latency.
• Familiarity with data serialization formats (JSON, Avro, Protobuf) and schema evolution.
• Experience in real-time data processing with Kafka Streams and writing KSQL queries.
• Experience deploying Kafka on cloud platforms (AWS, GCP, or Azure).
• Understanding of concepts like leader election, fault tolerance, and distributed consensus.
• Familiarity with Kafka management tools, logs, and metrics for performance monitoring and issue resolution.
• Knowledge of Kafka security features (authentication, authorization, encryption).
• Experience working in Agile environments, with the ability to work independently and lead efforts with minimal supervision.

Additional Skills

• Hands-on experience with Kafka Connect, KSQL, and Kafka Db connectors (e.g., Oracle, MySQL).
• Strong understanding of Kafka clustering, high availability (HA), and disaster recovery (DR) models.
• Ability to work in a fast-paced, dynamic environment and adapt to changing requirements.
• Strong problem-solving and debugging skills, with a passion for delivering high-quality, maintainable code.
• Experience with Confluent Control Center or Kafka monitoring tools (UI).

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 3+ years of experience in Kafka development, administration, and troubleshooting.

Posted By