Sr. Big Data Engineer

TechVilla Solutions

San Bruno, CA

Posted On: Aug 08, 2024

Posted On: Aug 08, 2024

Job Overview

Job Type

Contract - Corp-to-Corp, Contract - Independent, Contract - W2

Experience

8 - 12 Years

Salary

$70 - $80 Per Hour

Work Arrangement

On-Site

Travel Requirement

0%

Required Skills

  • SQL
  • Java
  • ETL
  • AWS
  • Hadoop
  • Apache
  • PostgreSQL
  • MongoDB
Job Description
Key Responsibilities
  • Design, develop, and maintain robust and scalable big data architectures, including data pipelines and ETL (Extract, Transform, Load) processes.
  • Deploy machine learning models to production environments.
  • Optimize existing data processing and storage solutions for performance, efficiency, and cost-effectiveness.
  • Collaborate with data scientists and analysts to understand data requirements and structure, ensuring the availability of high-quality data for analysis.
  • Implement and maintain data ingestion frameworks, using technologies such as Apache Kafka, Apache Spark, and Hadoop ecosystem tools.
  • Design and implement data models, database structures, and data warehousing solutions that support analytical workloads.
  • Monitor performance of big data systems, troubleshoot issues, and ensure high availability and reliability.
  • Code defensively, write unit tests, and ensure proper documentation of data architecture and related processes.

 

Qualifications
  • Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related field. Master’s degree preferred.
  • 8+ years of experience in data engineering, with a focus on big data technologies and architectures.
  • Proficiency in programming languages such as Java, Scala, or Python for data processing applications.
  • Strong hands-on experience with big data technologies such as Apache Hadoop, Apache Spark, Apache Hive, and Apache Kafka.
  • Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and their big data services (e.g., Amazon EMR, Azure Data Lake, Google BigQuery).
  • Familiarity with data storage systems, including relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra).
  • Understanding of data modeling concepts, data warehousing principles, and data governance best practices.
  • Knowledge of machine learning frameworks and tools (e.g., TensorFlow, PyTorch) is a plus.

Job ID: TS240328


Posted By

Vivek

Information Technology Recruiter