Research Engineer, Evals

ABOUT THE ROLE Build evaluation systems for a web data extraction product that turns pages into structured, LLM-ready information. You will shape how quality is measured, create the infrastructure to measure it at scale, and connect evaluation results to model and product decisions. WHAT YOU'LL DO - Design metrics for extraction quality across diverse websites, formats, and edge cases. - Build scalable pipelines and evaluation harnesses to measure output quality rigorously. - Generate and curate ground-truth datasets for trustworthy evaluation. - Close the feedback loop by translating evaluation results into model and product improvements. - Make quality results clear and actionable for the broader team. WHAT WE'RE LOOKING FOR - Four or more years of relevant engineering experience, including at least two years focused on evaluation work. - Experience building evaluations for LLM extraction of unstructured data, measuring whether fields are correctly extracted from messy, real-world pages or documents. - Experience building or evaluating large-scale web or document extraction systems and developing evaluation metrics and systems from the ground up. - Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, and evaluation frameworks. - Statistical rigor in metric design and interpretation, including sampling, significance, estimator bias and variance, error bounds, and judge-to-human agreement. - Experience creating ground-truth datasets and using evaluation findings to inform real model or product decisions. - A combination of research methodology and hands-on engineering, plus comfort working through ambiguous problems. COMPENSATION & BENEFITS Base salary: $250,000 to $290,000 USD annually. Visa sponsorship is available. LOCATION Hybrid, based in San Francisco, California, or Toronto, Canada. The role requires working on-site five days a week.