Data Engineer (Senior) w/ PySpark

Job Description

We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this role, you will be responsible for designing, building, and maintaining robust data pipelines and architectures that drive our data-driven decisions. You will work closely with cross-functional teams to ensure data availability, quality, and accessibility for various business needs.

Key Responsibilities

• Design, implement, and optimize large-scale data pipelines using PySpark.
• Manage and maintain data in SQL-based big data columnar databases such as Apache Druid, Google BigQuery, Cassandra, SAP HANA, etc.
• Leverage Google Cloud Platform (GCP) services like BigQuery, Firestore, and Pub/Sub to support data infrastructure.
• Develop and orchestrate data workflows using Airflow, particularly within the GCP environment (Cloud Composer).
• Collaborate with data scientists, analysts, and other stakeholders to ensure the availability and reliability of data.
• Optimize and troubleshoot data systems for performance, scalability, and reliability.
• Document data processes, architectures, and pipelines for maintenance and scalability.

Qualifications

Requirements

• PySpark: Strong experience and proficiency in building and optimizing data pipelines.
• SQL-based Big Data Columnar Databases: Expertise in working with databases like Apache Druid, Google BigQuery, Cassandra, SAP HANA, etc.
• Google Cloud Platform (GCP): Hands-on experience with BigQuery, Firestore, Pub/Sub.
• Airflow: Proven experience with Airflow, specifically within GCP (Cloud Composer) for data orchestration.
• Communication Skills: Excellent verbal and written communication skills, with the ability to collaborate effectively across teams.
• Scala: (Nice to have) Familiarity or experience with Scala programming language.
• Firebase: (Nice to have) Experience with Firebase services.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• Minimum of 5+ years of experience in data engineering or a related field.
• Proven track record of working with large-scale data processing and distributed systems.
• Strong problem-solving skills and ability to work independently in a fast-paced environment.

Additional Information

If you are a passionate Data Engineer with a strong background in big data technologies and are eager to take on new challenges, we would love to hear from you! Apply now to join our team and contribute to cutting-edge data solutions.