Key Responsibilities

• Design, develop, and maintain robust and scalable big data architectures, including data pipelines and ETL (Extract, Transform, Load) processes.
• Deploy machine learning models to production environments.
• Optimize existing data processing and storage solutions for performance, efficiency, and cost-effectiveness.
• Collaborate with data scientists and analysts to understand data requirements and structure, ensuring the availability of high-quality data for analysis.
• Implement and maintain data ingestion frameworks, using technologies such as Apache Kafka, Apache Spark, and Hadoop ecosystem tools.
• Design and implement data models, database structures, and data warehousing solutions that support analytical workloads.
• Monitor performance of big data systems, troubleshoot issues, and ensure high availability and reliability.
• Code defensively, write unit tests, and ensure proper documentation of data architecture and related processes.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related field. Master’s degree preferred.
• 8+ years of experience in data engineering, with a focus on big data technologies and architectures.
• Proficiency in programming languages such as Java, Scala, or Python for data processing applications.
• Strong hands-on experience with big data technologies such as Apache Hadoop, Apache Spark, Apache Hive, and Apache Kafka.
• Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and their big data services (e.g., Amazon EMR, Azure Data Lake, Google BigQuery).
• Familiarity with data storage systems, including relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra).
• Understanding of data modeling concepts, data warehousing principles, and data governance best practices.
• Knowledge of machine learning frameworks and tools (e.g., TensorFlow, PyTorch) is a plus.

Posted By