Software Engineer — Compute Platform

We’re on a mission to democratize AI by building the definitive AI data development platform. The AI landscape has gone through incredible change between 2016, when Snorkel started as a research project in the Stanford AI Lab, to the generative AI breakthroughs of today. But one thing has remained constant: the data you use to build AI is the key to achieving differentiation, high performance, and production-ready systems. We work with some of the world’s largest organizations to empower scientists, engineers, financial experts, product creators, journalists, and more to build custom AI with their data faster than ever before. Excited to help us redefine how AI is built? Apply to be the newest Snorkeler!

As a Compute Platform Engineer, you will play a pivotal role in building the backbone of SnorkelFlow's infrastructure. Your focus will be on orchestration, MLOps, SDK maintainability, and other tooling development that will serve our AI and Data teams. Whether you’re an entry-level engineer eager to dive into the complexities of compute engineering or a senior engineer ready to lead impactful projects, this role provides opportunities to work on cutting-edge AI infrastructure and scale SnorkelFlow’s capabilities.

Orchestration and Infrastructure Development:

• Design, implement, and maintain orchestration tools for workflows using Ray and Prefect.
• Build and manage infrastructure for scalable data connectors to integrate with systems like S3, Snowflake, and Databricks.
• Optimize compute resource utilization for AI pipelines and ensure reliable, fault-tolerant execution of tasks.

MLOps and SDK Development:

• Develop and maintain robust CI/CD pipelines to support seamless model deployment and orchestration workflows.
• Enhance the SnorkelFlow SDK to provide user-friendly access to compute layer functionalities.
• Collaborate with AI Platform and Data Platform teams to ensure SDK usability and extensibility.

Backend Development:

• Build backend services to support compute operations, including job scheduling, resource allocation, and API integrations.
• Partner with the Application team to design APIs that enable seamless orchestration and workflow management.

Observability and Performance Optimization:

• Implement monitoring tools and dashboards to track the performance and health of compute resources.
• Define metrics and logging strategies to optimize orchestration pipelines and SDK efficiency.

For Senior Engineers:

• Lead the design and development of core components of the compute platform, including orchestration workflows and connectors.
• Mentor and guide junior engineers, fostering growth within the team.
• Identify and drive improvements in infrastructure scalability and reliability.
• Collaborate with cross-functional teams to align the compute platform with overall product goals.

• Bachelor’s degree in Computer Science, Software Engineering, or a related field.
• 1-2 years of experience in backend development or infrastructure engineering.
• Proficiency in Python and familiarity with frameworks like FastAPI or Flask.
• Basic understanding of orchestration tools and MLOps practices.
• Strong problem-solving skills and eagerness to learn and grow in the compute engineering domain.

Senior Engineers:

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or a related field.
• 4-6 years of experience in backend or infrastructure engineering, including experience with orchestration tools (e.g., Prefect, Ray, Airflow).
• Proven expertise in Python, with experience in building scalable APIs and SDKs.
• Strong knowledge of MLOps practices, including CI/CD, Kubernetes, and model lifecycle management.
• Experience with infrastructure scaling and integration of data connectors.
• Proven ability to lead technical projects and mentor junior team members.

Preferred Qualifications

• Experience with Prefect or similar for orchestration and distributed computing.
• Experience with developing microservices pipelines and infrastructure
• Familiarity with vector databases and data storage solutions.
• Knowledge of containerization and orchestration tools like Docker and Kubernetes.
• Experience building data connectors for systems like Snowflake, Databricks, or S3.
• Familiarity with observability tools like Prometheus, Grafana, DataDog or OpenTelemetry.

What We Offer

• A leadership role with the opportunity to influence the technical direction of SnorkelFlow.
• Competitive salary and benefits tailored to your experience.
• Hybrid work environment with 3 days per week at our Redwood City HQ and SF Office.
• "No Meeting" Tuesdays and Thursdays to focus on deep work.
• The chance to work on cutting-edge infrastructure and drive impactful change in an innovative, fast-paced environment.

Be Your Best At Snorkel

Snorkel AI is on a mission to make machine learning practical for everyone, and it starts with building a team that welcomes, represents and gives opportunity to all. We work at the frontier of AI and software engineering, and believe that underrepresented communities need to play a part in shaping the future of these fields. At Snorkel AI, we actively work to create an environment that values end-to-end ownership, diverse forms of impact, and opportunities for personal growth.

Snorkelers are supported by an amazing team and an amazing set of benefits. For Full-time employees, we offer comprehensive medical, dental, and vision plans for Snorkelers and their families, plus a yearly wellness stipend. Our 401k program lets Snorkelers plan for their future and our parental leave program lets new parents take up to 20 weeks of paid time off. Learn more about these benefits and more — like our workstation setup allowance — on our Careers page.

Snorkel AI is proud to be an Equal Employment Opportunity employer and is committed to building a team that represents a variety of backgrounds, perspectives, and skills. Snorkel AI embraces diversity and provides equal employment opportunities to all employees and applicants for employment. Snorkel AI prohibits discrimination and harassment of any type on the basis of race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local law. All employment is decided on the basis of qualifications, performance, merit, and business need.

We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.