Must Have Technical/Functional Skills
• 10+ years of experience in Data Engineering, Data Warehousing, or Big Data development.
• 5+ years of hands-on experience with Databricks, Python, and PySpark.
• Strong hands-on experience in Databricks, Python, and PySpark development.
• Experience building and supporting large-scale ETL/ELT data pipelines.
• Strong knowledge of Spark SQL, Delta Lake, and Data Lakehouse architecture.
• Experience working with structured and unstructured data from multiple sources.
• Strong SQL skills and experience with relational databases.
• Experience with data modeling, data transformation, and data quality processes.
• Knowledge of orchestration tools such as Airflow, ADF, or similar.
• Experience with Git, CI/CD, and Agile development methodologies.
• Strong analytical, debugging, and performance tuning skills.
• Excellent verbal and written communication skills.
Roles Responsibilities
• Design, develop, and maintain data pipelines using Databricks, Python, and PySpark.
• Build scalable data ingestion, transformation, and integration solutions.
• Collaborate with Data Architects, Analysts, and business teams to deliver data solutions.
• Develop and optimize batch and real-time data processing workflows.
• Ensure data quality, integrity, security, and governance standards are met.
• Monitor, troubleshoot, and resolve production data pipeline issues.
• Optimize Spark jobs, SQL queries, and Databricks workloads for performance.
• Participate in data modeling, code reviews, and platform enhancement activities.
• Create technical documentation and support production deployments.
• Participate in Agile ceremonies and contribute to continuous improvement initiatives.