Spark Engineer

Required Skills

• Strong hands-on experience with Apache Spark 3.5.x or later.
• Expertise in Spark DataFrames, Datasets, RDDs, and Spark SQL.
• Strong understanding of Spark architecture, execution model, and distributed computing principles.
• Experience designing and building scalable ETL pipelines for batch and real-time data processing.
• Proven experience in Spark job tuning and performance optimization.
• Hands-on experience with cloud platforms such as AWS and/or Azure.
• Experience with cloud storage solutions including Amazon S3 and Azure Data Lake Storage (ADLS).
• Strong SQL and data modeling skills.
• Experience working in Linux/Unix environments.

Preferred Skills

• Experience with Snowflake for data warehousing and analytics.
• Exposure to streaming/real-time data processing frameworks.
• Familiarity with large-scale distributed data architectures.
• Knowledge of Scala programming language.

Roles & Responsibilities

• Design, develop, and maintain scalable batch and real-time data processing solutions using Apache Spark.
• Build and optimize ETL pipelines to support analytics and downstream applications.
• Develop high-performance data transformations using Spark SQL, DataFrames, Datasets, and RDDs.
• Perform Spark performance tuning, query optimization, and resource management for scalability and efficiency.
• Work with cloud-based storage and processing platforms in AWS/Azure environments.
• Collaborate with data engineers, architects, and analytics teams to deliver end-to-end data solutions.
• Ensure data quality, reliability, and adherence to engineering best practices.
• Support deployment, monitoring, and troubleshooting of Spark applications in distributed environments.

Required Qualifications

• Strong hands-on experience in Apache Spark and distributed data processing.
• Deep understanding of Spark internals, architecture, and execution lifecycle.
• Expertise in SQL-based data transformation and analysis.
• Experience with ETL frameworks, data modeling, and cloud-based data platforms.
• Good understanding of distributed systems and scalable data architectures.

Posted By