Research Scientist

ABOUT THE ROLE Join a research and engineering team developing benchmarks and evaluation methods for advanced language models. Your work will help AI labs and enterprise teams assess model capabilities and make better-informed decisions about production use. WHAT YOU'LL DO - Evaluate newly released AI models across a range of use cases. - Build benchmarks from the ground up, including dataset development and labeler coordination. - Develop scalable and accurate automated evaluation methods for generated text. - Work with engineering teams to implement and scale evaluation systems. - Collaborate with AI labs and enterprise teams to translate evaluation needs into research projects. WHAT WE'RE LOOKING FOR - At least 3 years of full-time AI/ML research experience, particularly in benchmarking, evaluation, or language models. - A track record of published research, ideally in NLP or benchmarking, and hands-on experience creating benchmarks or evaluation methods. - Strong Python skills and experience building clean, maintainable software; Django, Flask, or similar Python HTTP server experience is useful. - Experience with PyTorch or TensorFlow, language modeling, LLM infrastructure, and relevant deep learning techniques. - Strong computer science fundamentals and the ability to turn research findings into practical systems. - Experience collaborating through team development workflows, Git, code review, and feedback. - A degree in Computer Science, Machine Learning, or a related field; a Master's or PhD is preferred. Candidates with a Bachelor's degree and 1 to 3 years of applied experience may also be considered. COMPENSATION & BENEFITS Base salary: $150,000 to $200,000 annually. Visa sponsorship is available. LOCATION On-site in San Francisco, California.