We are seeking a Senior/Lead Databricks Developer with strong Snowflake experience and extensive ETL expertise. In this role, you will design and develop data solutions using Databricks, leveraging your knowledge of Scala and data engineering best practices to ensure efficient data ingestion and transformation.
Responsibilities
- Lead the design and development of data solutions in Databricks, primarily using Scala, focusing on end-to-end data ingestion and transformation.
- Utilize Snowflake for data warehousing and analytics, ensuring integration with existing Databricks workflows.
- Implement robust data engineering practices to build scalable data pipelines and workflows, optimizing for performance and reliability.
- Develop and maintain ETL processes, applying best practices for data warehousing, including various types of Slowly Changing Dimensions (SCD).
- Design and manage data lakes and Delta tables to support data storage and processing needs.
- Work with SQL Server and Oracle databases to facilitate data integration and reporting requirements.
- Collaborate with cross-functional teams to gather requirements and translate them into technical specifications for data solutions.
- Monitor, troubleshoot, and optimize data workflows to enhance system performance and efficiency.
Required Experience
- Proven experience as a Senior/Lead Developer with strong expertise in Databricks; familiarity with Snowflake is essential.
- Extensive background in data engineering and ETL processes.
- Proficient in Scala programming for data development within Databricks.
- Strong understanding of data warehousing concepts, including ETL best practices and SCD types.
- Knowledge of data lake architecture and Delta table functionality.
- Experience with SQL Server and Oracle databases.
- Experience with Hadoop and other big data technologies is a plus.
- Excellent analytical and troubleshooting skills, with a focus on optimizing data workflows.