Responsibilities

• Design, develop, and execute data pipelines and test cases to ensure data integrity and quality
• Develop and optimize ETL workflows integrating Amazon S3 for scalable data storage, retrieval, and processing
• Build and maintain scripts for automated data movement between S3 and other platforms using PySpark, Shell, Python, or similar languages
• Leverage Amazon S3 advanced features including versioning, lifecycle policies, access controls, and server-side encryption
• Collaborate with cross-functional teams to troubleshoot and resolve data-related issues in S3 environments
• Validate HiveQL, HDFS file structures, and data processing within the Hadoop cluster
• Maintain technical documentation and follow best practices for data storage, governance, and security
• Ensure robust metadata-driven ETL processes and batch/job framework support.

Required Skills

• Primary Skill: Amazon Data Engineer
• Secondary Skills: AWS Data Engineering, Amazon S3, Shell scripting, Autosys
• Strong experience with:
  • PySpark, SQL
  • Hadoop/Spark ecosystems (Cloudera preferred)
  • Data warehousing and ETL/ELT
  • Data quality and transformation
• Strong analytical and troubleshooting capabilities
• Excellent communication and stakeholder collaboration skills

Preferred / Domain Knowledge

• Banking domain knowledge, Payments preferred
• Experience with metadata-dependent ETL processes
• Familiarity with AWS data services and cloud storage concepts

Posted By