4 min read
How we measure whether Pilot's answers are right
Every week, reviewers grade thousands of Pilot answers against their sources. How the evaluation works, what we count as an error, and the numbers from our last quarter.
Elise Moreau and Hector Salinas
On this page
“Accurate” is the word every AI company uses and almost none define. We’d rather show our working. This post describes how we measure the quality of Pilot’s answers, what we count as an error, and where we’re still falling short.
What we measure
We grade answers on three separate questions, because an answer can pass one and fail another.
- Is each claim supported? Every factual sentence should be backed by a source, and the source should say what the sentence says.
- Is the answer complete? It should include the information a careful researcher would consider essential, including evidence that points the other way.
- Is it honest about uncertainty? When sources disagree or are missing, the answer should say so rather than pick one and sound confident.
We report these separately. A single “accuracy score” would hide the trade-offs. An assistant can look very accurate by refusing to answer anything hard.
The question set
Each quarter we refresh a set of 4,000 questions in four groups:
- Web research: current events, market data and public policy, where sources change often
- Your files: questions about uploaded documents, from short memos to 200-page reports
- Mixed: questions that need both, such as comparing a contract with current regulation
- Traps: questions with a false premise, or where no reliable answer exists
The trap questions matter most to us. A good answer to what did the 2025 EU study say about four-day weeks?, when no such study exists, is “I couldn’t find that study”, not a plausible summary.
How grading works
Every answer is graded by two trained reviewers who don’t know which version of Pilot produced it. They open each citation, find the passage, and mark each claim as supported, partly supported, unsupported or contradicted.
When the two reviewers disagree, a third settles it. Disagreement itself is useful: it usually means our grading guidelines are unclear, and we rewrite them.
We also use automated checks, but only to triage. A model flags answers that look risky so reviewers see them first. It never assigns a final grade.
Last quarter’s numbers
These are the results for the version that shipped in July, on the full question set.
| Measure | Web | Your files | Mixed | Traps |
|---|---|---|---|---|
| Claims fully supported | 93.8% | 96.1% | 90.4% | n/a |
| Claims contradicted by source | 0.9% | 0.4% | 1.6% | n/a |
| Answers judged complete | 84% | 91% | 78% | n/a |
| False premise caught | n/a | n/a | n/a | 87% |
Two things stand out. Answers about your own files are the most reliable, because the sources are fixed and well structured. Mixed questions are the hardest, because they require the model to reconcile two kinds of source with different dates and different levels of authority.
What counts as an error
We’re strict about this. An answer fails the support test if any of these happen:
- A number, date or name differs from the source
- The claim generalises beyond the source, such as a regional figure presented as global
- A qualifier in the source, such as “preliminary” or “estimated”, is dropped
- The citation points to the right document but the wrong passage
That last one surprises people. If the claim happens to be true but the citation doesn’t support it, a reader who checks will lose trust, and they’d be right to.
What we do with the results
Every contradicted claim is reviewed by an engineer and tagged with a cause: retrieval, reasoning, source quality or citation placement. Each quarter we pick the largest cause and work on it.
This year that meant long documents. Retrieval errors on reports over 40 pages were our biggest category, which led to the new Library index described in Long documents, short answers.
Where we’re still weak
We’d be doing exactly what we criticise if we only published the good numbers. Mixed questions are still too often incomplete. Answers in languages other than English are about two points less supported. And our trap set is written by our own team, which means it probably misses traps we haven’t thought of.
We’ll publish these numbers every quarter. If you’d like to see the guidelines our reviewers use, they’re available to Enterprise customers on request.

Written by
Elise Moreau
Head of Research
Leads the team that measures whether Pilot's answers are actually right.

With
Hector Salinas
Co-founder and CTO
Built search infrastructure at two companies before deciding answers need sources.
