Roles and Responsibilities

• Design, build, and maintain scalable and high-performance data pipelines for processing large volumes of data.
• Work with Hadoop ecosystem tools (HDFS, Hive, Pig, etc.) and PySpark to process and analyze large datasets efficiently.
• Develop and optimize data models and ETL workflows for data warehousing solutions, ensuring high-quality, accurate, and accessible data for reporting and analytics.
• Implement data job scheduling, automation, and monitoring using tools like Autosys.
• Work closely with data scientists, analysts, and business stakeholders to understand data requirements and provide data-driven solutions.
• Write complex SQL queries for data extraction, transformation, and loading (ETL), and use Python for data manipulation and automation tasks.
• Troubleshoot performance issues in data processing systems and optimize existing workflows for efficiency and scalability.
• Maintain clear and detailed documentation of data pipelines, processes, and system configurations.

Required Qualifications

• 7+ years of professional experience as a Data Engineer, with a strong focus on Big Data technologies and data warehousing.
• Expertise in Hadoop ecosystem (HDFS, Hive, Pig, etc.), and PySpark for distributed data processing.
• Proficient in designing and building ETL workflows using various tools and technologies.
• Proficiency in relational databases and SQL (e.g., Teradata, Oracle, PostgreSQL).
• Strong Python programming skills for automating tasks and processing data.
• Experience with job schedulers like Autosys for automating batch data processing workflows.

Posted By