Roles and Responsibilities

• Design, develop, and maintain Hadoop applications to analyze and process large data collections.
• Develop Hive queries and optimize Hive data warehousing solutions.
• Implement SQL and Unix scripting to support data processing workflows.
• Apply partitioning concepts and handle Slowly Changing Dimensions (SCD) in data processing.
• Utilize Spark frameworks including RDDs, DataFrames, and Datasets for efficient data transformations.
• Develop and maintain Shell scripts to automate and support batch processing jobs.
• Work with Yarn resource management for efficient job scheduling.
• Implement Kerberos authentication to ensure secure data access and cluster management.
• Troubleshoot and debug application issues and performance bottlenecks.
• Create and maintain data tracking and logging programs to ensure data integrity.
• Collaborate with cross-functional teams to understand business requirements and translate them into scalable data solutions.
• Keep up-to-date with new Hadoop ecosystem tools and technologies to recommend improvements.
• Work with SQL Server and Linux environments where applicable.

Required Skills and Qualifications

• Minimum 7 years of hands-on experience as a Hadoop Developer.
• Strong expertise in Hadoop ecosystem components, particularly Hive and Spark.
• Proficient in Hive query development and optimization.
• Experience with Spark concepts such as RDD, DataFrame, and Datasets.
• Good knowledge of SQL and Unix shell scripting.
• Understanding of partitioning strategies and Slowly Changing Dimensions.
• Familiarity with Yarn resource manager and Kerberos authentication.
• Strong problem-solving skills with the ability to troubleshoot complex issues.
• Knowledge of Linux environments and SQL Server is a plus.
• High-level analytical skills and attention to detail.

Posted By