Key Responsibilities

• Design and implement scalable data architectures on Azure Databricks, developing efficient data pipelines with PySpark for batch and stream processing.
• Develop and optimize data models (logical, physical, and conceptual) using industry-standard tools and frameworks to support data warehousing, data lakes, and business intelligence.
• Implement data governance practices, leveraging Unity Catalog in Databricks to manage data assets, enforce access controls, and ensure data integrity and compliance.
• Collaborate with business and technical teams to gather data requirements, provide solutions, and contribute to design reviews and best practices.
• Optimize data pipeline and model performance, troubleshoot issues, and ensure the reliability and availability of data across systems.

Required Skills and Experience

• 8+ years of experience in data engineering, data architecture, or related fields.
• Strong hands-on experience with Databricks for building and managing data pipelines.
• Proficiency in PySpark for large-scale data processing.
• Extensive experience with Azure cloud services, including Azure Data Lake, Azure SQL, Azure Synapse Analytics, and Azure Blob Storage.
• Expertise in data modeling techniques and tools (e.g., ERwin, IBM InfoSphere, Microsoft Visio, or similar).
• Experience with data governance frameworks and tools such as Unity Catalog in Databricks.
• Strong knowledge of data security, compliance, and best practices.
• Familiarity with ETL/ELT processes, data integration, and data warehousing principles.
• Experience in Agile or Scrum development environments.

Posted By