About the role
Every answer Pilot gives comes with sources. Your job is to know, with numbers, how often those sources support what Pilot says, and to make that number go up. You’ll design evaluations, build datasets with expert annotators and work with engineering to ship the improvements.
What you’ll do
- Design evaluations for citation accuracy, faithfulness and usefulness
- Build and maintain datasets with domain experts in law, finance and research
- Run experiments on retrieval, prompting and model choice
- Publish internal reports, and occasionally external ones, on what works
What we’re looking for
- A track record in NLP, information retrieval or evaluation methodology
- Strong Python and statistics, and comfort with messy real-world data
- The habit of writing down what you expect before you run an experiment
Nice to have
- Experience with human annotation programmes
- Publications on retrieval-augmented generation or factuality