The Role
The Lead Data Engineer will design and maintain a data lake and warehouse, build robust data pipelines for processing terabytes of data, mentor best practices in data architecture, and develop a long-term efficient data platform architecture.
Summary Generated by Built In
We are looking for a Data Engineering Tech lead to help us build state-of-the-art data infrastructure that will empower the most high-quality maps in the region.
Key Responsibilities:
- Design, build, and maintain Hudhud data lake and data warehouse
- Design and build reliable robust and efficient data pipelines using modern distributed processing technologies to process terabytes of data collected from various sources, including backend events, third-party data sources, and customer interactions across multiple channels.
- Foster, mentor, and enforce industry best practices in data architecture design, design patterns and coding standard.
- Solve most challenging data integration problems, utilizing optimal ETL patterns, frameworks, and query techniques, handling both structured and unstructured data sources
- Develop strategy for long-term Data Platform architecture to be efficient, reliable, and scalable
Basic Qualifications:
- BS or advanced degree in Computer Science, or related technical field.
- 6+ years of software development experience with a deep understanding of algorithms, data structures, data pipelines, and data warehousing.
- Strong hands-on experience in one or more programming languages such as Java, Scala, Python or similar languages.
- Proven background in developing distributed batch/streaming data pipelines (e.g. Spark, Kafka/Flink) using distributed storage systems (e.g., HDFS, S3)
- Comfortable navigating ambiguity and ownership of problem definitions in an agile environment.
- Excellent collaboration and communication abilities, with the ability to work effectively with cross-functional teams.
- Hands-on experience with relational databases (e.g., PostgreSQL, MySQL) and columnar databases (e.g., Redshift, BigQuery, HBase)
Preferred Qualifications:
- Familiarity with ETL schedulers such as Apache Airflow, Luigi, Oozie, AWS Glue or similar frameworks.
- Knowledge of container services (Docker/Kubernetes) is a plus.
- Experience working on/with end-to-end Machine Learning products is a significant plus.
- Experience in coaching and mentoring team members to foster professional growth and ensure best practices and coding standards are applied.
- Strong understanding of data architecture patterns.
Top Skills
Java
Python
Scala
The Company
What We Do
Join us on this journey, where every turn reveals something new and unexpected
