We are seeking a skilled and experienced Databricks Engineer to join our data engineering team. The ideal candidate will have hands-on experience with Databricks, including Unity Catalog, PySpark, and SQL, and will play a key role in designing, developing, and optimizing scalable data pipelines and analytics solutions.

Key Responsibilities

• Design and implement scalable data pipelines using Databricks, Delta Lake and Apache Spark (PySpark).
• Manage and govern data assets using Unity Catalog for secure and compliant data access.
• Write efficient and optimized SQL queries for data extraction, transformation, and reporting.
• Implement ETL processes to ingest, transform, and load data from various sources.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
• Monitor Databricks clusters, notebooks, jobs etc. , troubleshoot data workflows and performance issues.
• Ensure data quality, integrity, and security across all stages of the pipeline.
• Drive continuous improvement initiatives to enhance data processing efficiency
• Participate in monthly on-site meetings in Winston-Salem (3 days/month).

Required Qualifications

• 5+ years of experience in data engineering or analytics roles.
  
• Strong hands-on experience with Databricks, Delta Lake including Unity Catalog.
  
• Proficiency in PySpark and SQL.
  
• Experience with cloud platforms (Preferably AWS).
  
• Familiarity with data governance, access control, and metadata management.
  
• Excellent problem-solving and communication skills.
  
• Ability to travel to Winston-Salem once a month for 3 days.

Posted By