Primary DetailsTime Type: Full time
Worker Type: Employee
The purpose of this role is to drive the efficient delivery of build artefacts and provide technical leadership in the design, development, and deployment of QBE's platforms.
- Lead the design, development, and optimization of data pipelines and workflows within Azure Databricks using Python.
- Collaborate with the platform team to ensure proper alignment of resources, security protocols, and scalability for the Databricks environment.
- Architect, implement, and maintain optimized ETL pipelines, ensuring they are reliable, scalable, and efficient.
- Work closely with stakeholders to understand business requirements and design data solutions that meet their needs.
- Provide technical leadership to junior data engineers, guiding them in best practices for data pipeline design, data processing, and Databricks optimization.
- Coordinate with the platform team to ensure the availability, reliability, and performance of the underlying infrastructure.
- Lead troubleshooting and problem-solving efforts related to data processing and pipeline failures, ensuring minimal downtime.
- Monitor performance of data pipelines and implement optimization strategies to improve processing times, reduce costs, and enhance overall platform efficiency.
- Collaborate with data architects and engineers to develop data models that can be integrated seamlessly with Databricks.
- Maintain up-to-date knowledge of best practices and emerging technologies in data engineering, especially within the Azure ecosystem.
Requirements:
- 5+ years of experience in data engineering with a focus on building and optimizing data pipelines.
- Strong experience with Azure Databricks and the ability to lead and optimize complex workflows in this environment.
- Proficiency in Python for data processing, pipeline development, and automation.
- Experience with Azure Data Lake, Azure SQL, and other data storage and processing solutions within Azure.
- Deep understanding of ETL processes, including pipeline design, automation, and troubleshooting.
- Hands-on experience with Databricks Delta Lake, Apache Spark, and performance tuning in Databricks.
- Familiarity with cloud infrastructure (Azure), including networking, storage, and security considerations.
- Ability to collaborate and communicate effectively with cross-functional teams, including platform teams, to ensure alignment and successful delivery.
- Strong problem-solving and troubleshooting skills, particularly related to data pipeline performance and failures.
- Excellent communication and leadership skills, with the ability to mentor and guide junior engineers.
Preferred Qualifications:
- Experience with DevOps practices and CI/CD pipelines for data engineering workflows.
- Familiarity with Azure Synapse Analytics, Azure Data Factory, and other Azure-based data services.
- Experience working in an Agile environment with familiarity with project management tools.
Skills:
Communication, Critical Thinking, Customer Service, Financial Systems, Intentional collaboration, Managing performance, Process Improvements, Relational Database Management System (RDBMS), Risk Management, Software Development, Software Development Life Cycle (SDLC) Methodologies, Stakeholder Management, Strategic Leadership, Team Management, Waterfall Model
How to Apply:
To submit your application, click "Apply" and follow the step by step process.
Equal Employment Opportunity:
QBE is an equal opportunity employer and is required to comply with equal employment opportunity legislation in each jurisdiction it operates.
Top Skills
What We Do
QBE Insurance Group is one of the world's top 20 general insurance and reinsurance companies, with operations in all the key insurance markets. QBE is listed on the Australian Securities Exchange and is headquartered in Sydney. We employ more than 11,000 people in 27 countries.
