Lead Data Engineer

Job Description

We are seeking a highly skilled Azure Data Engineer to join our Advanced Analytics Data Engineering Team. The ideal candidate will possess a deep understanding of Azure Cloud, including Azure Data Factory (ADF), Azure Databricks, Unity Catalog, and other Azure resources. This role entails designing and building robust data pipelines, managing data storage solutions, and implementing data processing and analytics workflows that drive insights for decision-making. The candidate must be inquisitive of trends in Data Engineering domain like real time data processing, Data as a Product (DaaP), big data, data governance and security among others.

Responsibilities:

• Design and develop scalable and efficient data pipelines using Azure Data Factory (ADF) to ingest, clean, transform, and distribute data across various Azure services.
• Implement and maintain data lake architectures, ensuring data is structured for ease of access and analysis using services such as Azure Data Lake Storage.
• Leverage Azure Databricks to perform big data processing and analytics, including batch and stream processing, machine learning, and data exploration.
• Configure and manage Unity Catalog to organize data assets, enforce governance, and secure data access across Azure Databricks workspaces.
• Collaborate with business stakeholders to understand data requirements and deliver solutions that align with organizational goals and data strategies.
• Continuously optimize data engineering processes for performance, reliability, and cost-efficiency.
• Monitor and troubleshoot data pipelines to ensure seamless flow and availability of data.
• Keep oneself exposed to the latest Azure features and technologies, integrating them into solutions when appropriate.
• Ensure data security and compliance with relevant data protection regulations.
• Document all aspects of the data engineering process, from pipeline design to deployment and maintenance.
• Provide mentorship and technical guidance to junior data engineers and team members.

Pre-requisite skills & experience:

• Bachelor’s/master’s degree in computer science, Data Science, Information Technology, or related fields.
• Minimum of 6 years of experience in a Data Engineering role, with a focus on Azure data services.
• Expertise in Azure Data Factory (ADF) and designing ETL/ELT data pipelines.
• Proficiency in Azure Databricks for big data processing, analytics, and machine learning workflow implementations.
• Experience maintaining and extending production data systems to evolve with complex needs.
• Familiarity with Unity Catalog and its integration with Azure Databricks for data governance.
• Deep experience with distributed computing with Apache Spark and knowledge of Spark runtime internals
• Design and deployment of performant end-to-end data architectures
• Strong understanding of SQL, Python, Java and/or Scala for data manipulation and transformation.
• Experience with Azure services such as Azure Data Lake Storage, Azure Synapse Analytics, and Azure SQL Database.
• Knowledge of data modeling, warehousing, and data lake patterns and practices in Azure.
• Excellent problem-solving, analytical, and communication skills.
• Ability to work independently and collaboratively in a fast-paced and dynamic environment.
• Relevant Azure certifications are a plus (e.g., Azure Data Engineer Associate, Azure Data Scientist Associate).

Good to have skills

• CI/CD for Data Pipelines DevOps, DataOps or MLOps , Azure DevOps
• Knowledge of Azure Monitor, Azure Log Analytics, and other Azure tools for real-time monitoring, logging, and performance management of data engineering solutions.
• Expertise in handling real-time streaming data, Kafka and IoT (Internet of Things) data ingestion, processing, and analytics using Azure Stream Analytics, Azure IoT Hub, and related Azure services.