We are looking for a Databricks Tech Lead responsible for designing, building, and maintaining data pipelines that transform raw data into actionable insights within the Databricks environment. The ideal candidate will have expertise in Apache Spark, real-time data ingestion, data modeling, and cloud-native tools to optimize data flow and processing.
Roles & Responsibilities
- Design and implement data pipelines using Databricks, Apache Spark, and Python.
- Manage and optimize large datasets, data flow, and processing.
- Develop data models, storage, access, and analytics platforms.
- Implement database backup, recovery, and archiving strategies.
- Work with real-time data ingestion using StreamSets.
- Utilize UDM modeling, dimensional modeling, data migration, and ETL techniques.
- Provided solution design guidance and assisted in system integration.
- Collaborate with stakeholders to translate business requirements into technical designs.
- Work with Azure Data Lake, SQL Data Warehouse, Kubernetes, and Docker to build scalable solutions.
- Write SQL, Python, and PySpark for efficient data transformation.
- Develop mapping documents and transformation rules as per project scope.
- Communicate project status and ensure smooth execution.
Requirements
- Strong experience in Databricks, Python, Apache Spark, Kubernetes, and Docker.
- Hands-on expertise in data modeling, data migration, data cleansing, and ETL techniques.
- Knowledge of real-time data ingestion and cloud-native tools.
- Experience working with Azure stack modules (Data Lakes, SQL DW).
- Ability to design scalable and efficient data pipelines.
- Strong communication skills to collaborate with technical and business teams.