Duties and Responsibilities

• Design and develop a comprehensive big data architecture that meets the organization's needs for data storage, processing, and analysis.
  
• Work closely with stakeholders to understand business requirements and develop data solutions that meet those needs.
• Design and implement scalable, secure, and efficient data pipelines for data ingestion, processing, and storage.
• Collaborate with data scientists to develop data models and algorithms that drive business insights.
  
• Develop and maintain data quality standards and governance policies.
• Work with the engineering team to ensure that big data technologies are integrated with existing systems and infrastructure.
  
• Develop and maintain documentation of the big data architecture and its components.

Required Experience/skills

• Expertise in Cloudera Big Data Technologies, including Cloudera Distribution of Hadoop (CDP), Apache Hadoop, Apache Spark, Apache Hive, Apache Impala, and Apache HBase, with hands-on experience in deploying, configuring, and optimizing these technologies for large-scale data processing and analytics.
• Experience working with development tools like Jenkins for continuous integration, deployment, and pipeline automation, streamlining the software development lifecycle and ensuring rapid delivery of high-quality solutions.
• Hands-on experience with ELK (Elasticsearch, Logstash, Kibana) stack, including configuration, customization, and management of Kibana dashboards for real-time monitoring and visualization of system metrics and application logs.
• In-depth knowledge of data governance policies, metadata management, and regulatory compliance requirements, with a focus on protecting sensitive data and ensuring data privacy and security.
• Proficiency in Python for advanced data analysis, scripting, automation, and integration with Big Data platforms and ecosystems.
  
• Strong SQL experience (Oracle, Hive, Impala) in designing and implementing complex data models, DDLs, and DMLs, with a focus on performance optimization and scalability.

Posted By