Responsibilities

• Design, develop, and optimize ETL pipelines for large-scale data processing and transformation.
• Leverage Databricks tools and technologies, including Delta Lake and Databricks SQL, to manage and process data effectively.
• Implement real-time data processing solutions using Databricks Spark Streaming and Structured Streaming frameworks.
• Build scalable, distributed data workflows using PySpark and Spark SQL.
• Develop reliable and automated pipelines using Delta Live Tables.
• Utilize Autoloader for efficient incremental data ingestion.
• Troubleshoot and optimize performance in distributed computing environments.
• Collaborate with cross-functional teams to ensure data solutions align with business requirements.
• Maintain expertise in Azure data services and related technologies.

Qualifications

• Minimum of 12+ of hands-on experience in data engineering.
• Expertise in Databricks, including Delta Lake and Databricks SQL.
• Proficiency in ETL development, PySpark, and large-scale data workflows.
• Strong knowledge of streaming data pipelines and frameworks like Spark Structured Streaming.
• Familiarity with the Azure platform and its data services.
• Exceptional troubleshooting and performance optimization skills in distributed environments.

Posted By