Research

Research Scientist, Evaluation

Measure whether Pilot's answers are right, cited and useful, and turn what you learn into better models and prompts.

About the role

Every answer Pilot gives comes with sources. Your job is to know, with numbers, how often those sources support what Pilot says, and to make that number go up. You’ll design evaluations, build datasets with expert annotators and work with engineering to ship the improvements.

What you’ll do

  • Design evaluations for citation accuracy, faithfulness and usefulness
  • Build and maintain datasets with domain experts in law, finance and research
  • Run experiments on retrieval, prompting and model choice
  • Publish internal reports, and occasionally external ones, on what works

What we’re looking for

  • A track record in NLP, information retrieval or evaluation methodology
  • Strong Python and statistics, and comfort with messy real-world data
  • The habit of writing down what you expect before you run an experiment

Nice to have

  • Experience with human annotation programmes
  • Publications on retrieval-augmented generation or factuality

Other open roles

Build the tools people write with

Remote first, small teams, and work that ships every week.