We are seeking a skilled Kafka Architect to design, implement, and optimize real-time data streaming solutions using Apache Kafka, KSQL, and Confluent Cloud. The ideal candidate will lead the integration of Kafka into various applications, ensuring scalability, performance, and seamless data flow across systems.
Key Responsibilities
- Design, deploy, and manage scalable, high-availability Kafka clusters in both on-prem and cloud environments (Confluent Cloud).
- Use KSQL for real-time data transformation and stream processing, designing data pipelines and event-driven architectures.
- Integrate Kafka with internal systems, third-party services, and databases for smooth data flow and real-time application updates.
- Continuously monitor and optimize Kafka performance (throughput, latency, resource utilization) and plan for future capacity needs.
- Implement security measures such as encryption, access control, and ensure compliance with data governance standards.
- Work closely with data engineers, developers, and DevOps teams to ensure Kafka is integrated and operates efficiently within the organization’s ecosystem.
- Maintain technical documentation and promote Kafka best practices across teams.
Required Qualifications
- 7+ years in distributed systems, with expertise in Apache Kafka, KSQL, and Confluent Cloud.
- Proficient in Kafka architecture, KSQL, stream processing, and cloud platforms (AWS, GCP, Azure). Experience with Kafka Connect, microservices, and application integration.
- Strong skills in Java, Scala, or Python for building Kafka producers/consumers.
- Experience with CI/CD, Docker, Kubernetes, and managing Kafka in cloud environments.
Preferred Qualifications
- Confluent Certified Developer for Apache Kafka or cloud platform certifications.
- Familiarity with other streaming platforms (e.g., Flink, Pulsar) and big data tools (e.g., Spark, Hadoop).