Senior Technical Product Manager, ML Quality

Who We Are: WellSaid 

WellSaid is the leading AI voice company for enterprise and professional applications. Using carefully sourced voice talent and our own AI advanced platform, WellSaid provides ultra-realistic voices that the world’s biggest brands trust to engage listeners. We build AI responsibly and ethically. 
Who You Are: An Experienced Technical Product Manager with a Focus on ML & Model Performance

You are an experienced product manager capable of managing the requirements, timelines, and overall roadmap of technical products in generative AI quality assurance. You have a strong technical background and a history of collaborating successfully in ML, TTS, or a related field. More specifically, in your previous role(s) you have been responsible for overseeing the testing and validation of ML models to guarantee their accuracy, reliability, and ethical implementation within a product.  Ideally, you have experience in audio and can manage technical conversations pertaining to audio signal qualities and methods for measuring and identifying them. 

You manage product delivery through the R&D flywheel: from research to development, to testing, to delivery, to feedback, and back to research. You write clear project plans and define requirements in a way that is comprehensive and can be handed off to engineers to design and develop against.  You rely on your research skills to inform your product requirements, test case suites, and testing plans. You work well with data engineers and analysts to make informed recommendations and evaluation reports. You are able to draw connections between model limitations and methods for measuring them.

Further, you take the ethics surrounding AI very seriously and work hard to protect all stakeholders in data gathering, research, and exposure. You are a creative thinker and strong collaborator who enjoys working with engineering teams to build effective solutions with a meaningful impact on our products. 

How You’ll Contribute:  

As Senior Technical Product Manager, ML Quality, you will manage the development of products and processes that are used to evaluate data, models, and features developed by the ML team. Your evaluations may result in reports highlighting major model developments to the broader WS community or in guidance for the development team honing various TTS features. For example, you might ensure the quality and accuracy of new TTS models before they launch to customers through data research, model testing, and regression analysis.
You will be responsible for:

• Defining QA strategy for ML models across many languages: Identifying critical metrics and testing methodologies to assess the performance of TTS models throughout the research & development lifecycle. This includes developing such strategies for any new languages we explore in the future.
• Evaluating and validating TTS models: Working with the Applied ML team and selecting appropriate evaluation metrics, running rigorous tests across datasets, experiments, and final models, and interpreting results to identify areas for improvement and/or offer recommendations to the Research team. 
• Managing data quality:  Monitoring data quality, identifying potential biases in training data, and ensuring data preprocessing is done correctly to optimize model performance. 
• Developing QA automation frameworks: Defining automated testing tools and pipelines to streamline the testing and evaluation processes for TTS models, especially for regression testing. 
• Monitoring model performance in production: Tracking key metrics post-deployment to identify potential degradation in model performance and take corrective action. This may also require defining the monitoring and tooling needed to do so.
• Moderating ethical considerations: Assessing potential biases in ML models and training data, ensuring responsible development practices, and addressing concerns regarding fairness, transparency, and Responsible AI. 
• Researching and partnering with third parties: Defining test cases, scenarios, and evaluation batches with external evaluation services when appropriate. Recommending evaluation services to leadership with a well-defined testing strategy defending your recommendation.

In your day-to-day, you will:  

• Collaborate closely with data engineers and software engineers in the ML team to understand model capabilities, identify new methods or metrics for gauging model progression, and create new tools to make the testing and evaluation processes fast and easy to implement
• Write design docs with clear requirements and expectations for engineers to execute against
• Research and collate text and audio test cases from real customer data, product feedback, and active experiments.
• Synthesize evaluation results into reports for internal and external publication and/or deliver recommendations to the research team informed by rigorous regression analysis
• Partner closely with Lead Product Manager, ML on overall product roadmap, customer impact, and development recommendations
  

What We’re Looking For 

To thrive in this role,  you also have some combination of the following: 

• 2+ Machine Learning Product Management experience  at a similar-stage company
• 5+ years in ML, TTS, or related fields with strong instincts for identifying market differentiators
• Tenured understanding and experience with ML research, testing, and deployment
• Experience in audio engineering, analysis, or related field
• Strong cross-team collaboration skills, especially collaborating with ML and Data Engineers
• Experience managing multiple projects at once with a track record of predictable, on-time, on-budget delivery
• Clear, direct communication skills (written and verbal)
• Organization skills that demonstrate you are process-oriented
• Strong ability to create order from ambiguity
• Kind, empathetic, and intentional leadership qualities
• Previous ML engineering experience is a plus!

If you have experience in audio signal processing or linguistic support at the machine learning level, and some of the qualifications above, we strongly encourage you to apply.

To join our team you also:

• Must be a U.S. Citizen or Permanent Resident
• Must pass a pre-employment background check

What We Offer 

WellSaid is proud to support an inclusive work environment that emphasizes each team member’s personal and professional growth. Our team is headquartered in Bellevue, Washington, with members distributed throughout the U.S.

As a technology company, we strive to be externally competitive with companies of a similar size and stage, and internally fair in our pay practices. The hiring salary range for this role is $130,000-$170,000 annually and represents the target offer range given the scope and experience expectations for this role.

Other perks and benefits: 

• Competitive salary and stock options
• Full medical, dental, and vision insurance
• Matching 401(k) plan
• Generous vacation policy/paid time off
• Parental leave
• Learning & development stipend
• Home office stipend
  

What to Expect From Us 

We encourage you to apply! If we feel your skills, experience, and values match, we’ll reach out about meeting with the team. 

During the interview stage, you can expect:

• An initial “phone” screen with our recruiting team (45 minutes); if there’s a match we’ll invite you to meet further with the team.
• An initial meeting with our Lead Product Manager, Machine Learning (45 minutes). 
• An Interview loop with 3-4 interviews (1 hour each) with the team members you will be working most closely with.

All interviews will be remote via Google Meets; we are happy to make accommodations you might need to feel set up for success in our process.