We are seeking a highly skilled Databricks DevOps Lead to drive automation, CI/CD, and infrastructure management for our data platforms on Azure. The ideal candidate will have deep experience with Databricks, Spark, Azure services, and DevOps practices, enabling seamless deployment, monitoring, and operational excellence of distributed data systems. This role also includes coordinating with offshore teams and mentoring junior engineers.
Mandatory Skills
- Databricks and Apache Spark development & deployment
- Azure cloud services: Cosmos DB, ADLS Gen2, Event Hubs, Key Vault, AKS
- Infrastructure as Code (IAAS): Terraform, Bicep, ARM templates
- DevOps Tools: Azure DevOps (GitHub, CI/CD pipelines, Boards), GitHub Actions
- Scripting & Automation: Python, PowerShell, Bash
- Containerization & Orchestration: Docker, Azure Kubernetes Service (AKS)
- Monitoring & Observability: Grafana, Azure Monitor, Log Analytics
- Testing & Quality: JUnit, Postman, SonarQube
Preferred / Additional Skills
- 5+ years of DevOps or Site Reliability Engineering (SRE) experience supporting data platforms or distributed systems
- Strong expertise in CI/CD pipelines for Databricks, Spark jobs, and Azure data components
- Hands-on experience with provisioning Azure resources via IaC
- Knowledge of release strategies for notebooks, JARs, wheels, and configuration artifacts
- Understanding of security best practices (managed identities, secrets management, RBAC) for Azure data services
- Exposure to reliability engineering concepts such as error budgets, SLOs, and SLI dashboards
Roles & Responsibilities
- Design, implement, and maintain CI/CD pipelines for Databricks, Spark jobs, and Azure data services
- Develop and maintain Infrastructure as Code (Terraform/Bicep/ARM) to provision and manage Azure resources
- Build automation scripts for deployment and release management of notebooks, JARs, wheels, and configuration artifacts
- Implement monitoring, logging, and alerting frameworks for Spark jobs and data pipelines using Azure Monitor, Log Analytics, or Grafana
- Coordinate with offshore teams to ensure alignment on deployments, releases, and support activities
- Ensure adherence to DevOps best practices, security standards, and operational excellence
- Mentor and guide junior team members on DevOps practices, CI/CD pipelines, and infrastructure management