We are seeking an experienced Senior Data Quality Engineer to join our dynamic team. In this role, you will be responsible for ensuring the integrity, accuracy, and reliability of large datasets used for various analytics functions across the organization. You will work closely with data engineers, analysts, and stakeholders to build robust data pipelines, validate data quality, and establish frameworks for ongoing data validation and quality control.
Key Responsibilities
- Design, implement, and maintain data quality frameworks to ensure high standards for data integrity, accuracy, and consistency.
- Develop and execute complex SQL queries (including Spark SQL) on large datasets to identify, analyze, and resolve data issues.
- Use Databricks (in an AWS environment) to build and manage data processing workflows, ensuring data quality across systems.
- Develop Python and PySpark scripts to perform data validation, transformation, and other data quality checks.
- Collaborate with cross-functional teams to define data quality standards and ensure consistent implementation across all data sources.
- Build and optimize ETL/ELT processes to transform data from multiple sources, ensuring it is accurate and ready for analytical consumption.
- Design and develop automated testing and monitoring for data pipelines and transformations to catch data issues early in the process.
- Manage and enhance data pipelines in Databricks notebooks, ensuring high performance and reliability.
- Provide support for troubleshooting data quality issues and assist in root cause analysis to resolve recurring data problems.
- Continuously assess and improve data quality practices, proposing and implementing best practices for ongoing data governance.
Required Skills and Qualifications
- 5+ years of experience in data quality engineering, preferably in the data or analytics space.
- Strong proficiency with SQL and Spark SQL, including complex query writing and optimization on large datasets.
- Expertise with Databricks, particularly in an AWS environment.
- Proficiency in Python, PySpark, and Pandas for data manipulation, validation, and scripting.
- Experience with building and optimizing ETL/ELT pipelines for large-scale data transformations.
- Demonstrated ability to work with Databricks notebooks for data processing, testing, and pipeline development.
- Experience in developing and maintaining automated testing and data validation frameworks.
- Ability to manage and ensure the quality of data pipelines, ensuring reliability and consistency across systems.
Preferred Qualifications
- Experience working with Databricks in an AWS cloud environment.
- Familiarity with best practices for data governance, including data lineage, auditing, and traceability.