We are seeking an experienced Spark Admin with Hadoop expertise to join our team. As a Spark Admin, you will be responsible for managing and maintaining our Spark and Hadoop clusters, ensuring their optimal performance, availability, and security. You will work closely with cross-functional teams to support data processing and analysis requirements.

Responsibilities:

• Manage and administer Spark and Hadoop clusters, ensuring their smooth operation and optimal performance.
• Install, configure, and deploy Spark and Hadoop components, including HDFS, YARN, Hive, Spark SQL, and other related tools.
• Monitor cluster health, resource utilization, and job performance using cluster management and monitoring tools.
• Troubleshoot and resolve issues related to cluster performance, data processing, and job failures.
• Implement security measures and access controls to protect data and ensure compliance with data governance policies.
• Collaborate with data engineers and data scientists to optimize data processing workflows and job scheduling.
• Stay up to date with the latest Spark and Hadoop releases, patches, and best practices.
• Automate administrative tasks and develop scripts to improve efficiency and productivity.
• Provide technical support and guidance to development teams and end users.
• Conduct performance tuning and optimization of Spark and Hadoop configurations.

Qualifications:

• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
• Minimum of 5 years of experience as a Spark Admin, Hadoop Administrator, or a similar role.
• Strong proficiency in Spark and Hadoop administration, including installation, configuration, and maintenance.
• Experience with cluster management and monitoring tools such as Ambari, Cloudera Manager, or Hortonworks.
• Knowledge of Hadoop ecosystem components, such as HDFS, YARN, Hive, Spark SQL, and MapReduce.
• Familiarity with Linux/Unix environments and shell scripting.
• Understanding of networking, storage, and security concepts related to Spark and Hadoop clusters.
• Strong problem-solving and troubleshooting skills.
• Excellent communication and collaboration abilities.