We are seeking a hands-on SRE Engineer with expertise in Apache Spark, Kubernetes, CI/CD pipelines, Python, PL/SQL, and containerization to design, develop, and maintain real-time data processing and infrastructure. The ideal candidate will combine development skills with operational expertise to ensure high availability, scalability, and reliability of applications and infrastructure.
Roles and Responsibilities
- Provide technical leadership and guidance to engineering teams, participating in key technical decisions.
- Design, develop, and maintain real-time data pipelines and Data Lakehouse infrastructure.
- Write Python scripts for automation and small functionality development.
- Write SQL queries and procedures for data processing.
- Build and maintain CI/CD pipelines using Jenkins and related tools.
- Manage applications in Kubernetes environments, including deployment, configuration, access control, and troubleshooting.
- Develop and maintain Kubernetes manifests, Helm charts, and deployment artifacts.
- Manage Docker containers, including image management in private registries.
- Maintain Apache Spark clusters and related workflows.
- Develop and maintain Ansible playbooks for infrastructure configuration and management.
- Implement Infrastructure as Code (IaC) and collaborate with teams to ensure consistent infrastructure management.
- Monitor services and infrastructure using observability tools for capacity management and performance tuning.
- Drive initiatives to containerize standalone applications and migrate them to Kubernetes.
- Mentor and guide other engineers in development, testing, and deployment practices.
Technical Skills
- Programming & Scripting: Python, PL/SQL
- Data & Processing: Apache Spark, SQL
- CI/CD & Automation: Jenkins, Ansible, CI/CD pipelines
- Containerization & Orchestration: Docker, Kubernetes (EKS/ECS experience a plus)
- Infrastructure & Deployment: Helm charts, IaC
- Monitoring & Observability: Dynatrace or similar monitoring/tracing tools
- Data Platforms: Dremio experience is a plus