Director of AI Model Evaluation & Quality Validation - LLMs & Generative AI
Truveta is the world's first health provider led data platform with a vision of Saving Lives with Data. Our mission is to enable researchers to find cures faster, empower every clinician to be an expert, and help families make the most informed decisions about their care. Achieving Truveta's ambitious vision requires an incredible team of talented and inspired people with a special combination of health, software and big data experience who share our company values.
Who We Need
Truveta is rapidly building a talented and diverse team to tackle complex health and technical challenges. Beyond core capabilities, we are seeking problem solvers, passionate and collaborative teammates, and those willing to roll up their sleeves while making a difference. If you are interested in the opportunity to pursue purposeful work, join a mission-driven team, and build a rewarding career while having fun, Truveta may be the perfect fit for you.
This Opportunity
We are seeking a highly skilled and experienced Machine Learning Expert to join our team and evaluate the effectiveness of our AI models utilizing Large Language Models (LLMs). The ideal candidate will have a strong background in machine learning, natural language processing (NLP), and a proven track record of working with LLMs. You will be responsible for assessing the performance, accuracy, and efficiency of our AI models, identifying areas for improvement, and providing actionable insights to enhance their effectiveness. You will work in an exciting and fast-paced environment, collaborating closely with multiple teams across the company.
In This Role, You Will:
- Evaluate the performance of AI models that leverage Large Language Models (LLMs) in various applications.
- Develop and implement metrics to measure the accuracy, efficiency, and overall effectiveness of LLM-based AI models.
- Conduct thorough analyses to identify strengths and weaknesses in model performance.
- Provide detailed reports and recommendations to improve model accuracy, efficiency, and scalability.
- Collaborate with the development team to refine and enhance AI models based on evaluation results.
- Stay up-to-date with the latest advancements in machine learning, NLP, and LLMs to ensure our models remain at the forefront of the industry.
- Ensure that models meet business requirements and deliver tangible value to our clients.
Key Qualifications
- Master's or Ph.D. in Computer Science, Machine Learning, Artificial Intelligence, or a related field.
- 7+ years of experience in machine learning, with a focus on natural language processing and large language models.
- Proven track record of evaluating and improving AI models, particularly those involving LLMs.
- Strong programming skills in Python and experience with machine learning frameworks such as TensorFlow, PyTorch, or similar.
- Familiarity with performance evaluation metrics and techniques for LLMs.
- Excellent analytical and problem-solving skills.
- Ability to communicate complex technical concepts to non-technical stakeholders.
Preferred Qualifications
- Experience with popular LLMs such as GPT, BERT, T5, or similar.
- Knowledge of deployment and scaling of AI models in a production environment.
- Previous experience working in a fast-paced, agile development environment.
- Contributions to the AI/ML community through research papers, conference presentations, or open-source projects.
This position offers a unique chance to work on state-of-the-art language models and contribute to transformative research with the vision of Saving Lives with Data. You will be part of a diverse team who are passionate about pushing the boundaries of machine learning and natural language understanding in the healthcare domain. Join us and make a significant impact on the future of healthcare and patient well-being.
Why Truveta?
Be a part of building something special. Now is the perfect time to join Truveta. We have strong, established leadership with decades of success. We are well-funded. We are building a culture that prioritizes people and their passions across personal, professional, and everything in between. Join us as we build an amazing company together.
We Offer:
- Interesting and meaningful work for every career stage
- Great benefits package
- Comprehensive benefits with strong medical, dental and vision insurance plans
- 401K plan
- Professional development & training opportunities for continuous learning
- Work/life autonomy via flexible work hours and flexible paid time off
- Generous parental leave
- Regular team activities (virtual and in-person as soon as we are able)
- The base pay for this position is $230,000 to $260,000. The pay range reflects the minimum and maximum target. Pay is based on several factors including location and may vary depending on job-related knowledge, skills, and experience. Certain roles are eligible for additional compensation such as incentive pay and stock options.
Truveta is committed to creating a diverse, inclusive, and empowering workplace. We believe that having employees, interns, and contractors with diverse backgrounds enables Truveta to better meet our mission and serve patients and health communities around the world. We recognize that opportunities in technology historically excluded and continue to disproportionately exclude Black and Indigenous people, people of color, people from working class backgrounds, people with disabilities, and LGBTQIA+ people. We strongly encourage individuals with these identities to apply even if you don't meet all of the requirements.
