4 min read

How we measure whether Pilot's answers are right

Every week, reviewers grade thousands of Pilot answers against their sources. How the evaluation works, what we count as an error, and the numbers from our last quarter.

Elise Moreau and Hector Salinas

Research
On this page

“Accurate” is the word every AI company uses and almost none define. We’d rather show our working. This post describes how we measure the quality of Pilot’s answers, what we count as an error, and where we’re still falling short.

What we measure

We grade answers on three separate questions, because an answer can pass one and fail another.

  1. Is each claim supported? Every factual sentence should be backed by a source, and the source should say what the sentence says.
  2. Is the answer complete? It should include the information a careful researcher would consider essential, including evidence that points the other way.
  3. Is it honest about uncertainty? When sources disagree or are missing, the answer should say so rather than pick one and sound confident.

We report these separately. A single “accuracy score” would hide the trade-offs. An assistant can look very accurate by refusing to answer anything hard.

The question set

Each quarter we refresh a set of 4,000 questions in four groups:

  • Web research: current events, market data and public policy, where sources change often
  • Your files: questions about uploaded documents, from short memos to 200-page reports
  • Mixed: questions that need both, such as comparing a contract with current regulation
  • Traps: questions with a false premise, or where no reliable answer exists

The trap questions matter most to us. A good answer to what did the 2025 EU study say about four-day weeks?, when no such study exists, is “I couldn’t find that study”, not a plausible summary.

How grading works

Every answer is graded by two trained reviewers who don’t know which version of Pilot produced it. They open each citation, find the passage, and mark each claim as supported, partly supported, unsupported or contradicted.

When the two reviewers disagree, a third settles it. Disagreement itself is useful: it usually means our grading guidelines are unclear, and we rewrite them.

We also use automated checks, but only to triage. A model flags answers that look risky so reviewers see them first. It never assigns a final grade.

Last quarter’s numbers

These are the results for the version that shipped in July, on the full question set.

Measure Web Your files Mixed Traps
Claims fully supported 93.8% 96.1% 90.4% n/a
Claims contradicted by source 0.9% 0.4% 1.6% n/a
Answers judged complete 84% 91% 78% n/a
False premise caught n/a n/a n/a 87%

Two things stand out. Answers about your own files are the most reliable, because the sources are fixed and well structured. Mixed questions are the hardest, because they require the model to reconcile two kinds of source with different dates and different levels of authority.

What counts as an error

We’re strict about this. An answer fails the support test if any of these happen:

  • A number, date or name differs from the source
  • The claim generalises beyond the source, such as a regional figure presented as global
  • A qualifier in the source, such as “preliminary” or “estimated”, is dropped
  • The citation points to the right document but the wrong passage

That last one surprises people. If the claim happens to be true but the citation doesn’t support it, a reader who checks will lose trust, and they’d be right to.

What we do with the results

Every contradicted claim is reviewed by an engineer and tagged with a cause: retrieval, reasoning, source quality or citation placement. Each quarter we pick the largest cause and work on it.

This year that meant long documents. Retrieval errors on reports over 40 pages were our biggest category, which led to the new Library index described in Long documents, short answers.

Where we’re still weak

We’d be doing exactly what we criticise if we only published the good numbers. Mixed questions are still too often incomplete. Answers in languages other than English are about two points less supported. And our trap set is written by our own team, which means it probably misses traps we haven’t thought of.

We’ll publish these numbers every quarter. If you’d like to see the guidelines our reviewers use, they’re available to Enterprise customers on request.

evaluationcitationsaccuracy
Portrait of Elise Moreau

Written by

Elise Moreau

Head of Research

Leads the team that measures whether Pilot's answers are actually right.

Portrait of Hector Salinas

With

Hector Salinas

Co-founder and CTO

Built search infrastructure at two companies before deciding answers need sources.

All posts
  • Product

    4 min read

    Release 4.2: Autopilot agents

    Autopilot agents now write your recurring reports on a schedule, from the sources you choose, and wait for a person to approve them. Library search is also three times faster.

    Tomás Ferreira and Hector Salinas

  • Guides

    6 min read

    How to check an AI answer in under a minute

    A five-step routine for checking an AI answer before you use it: read the sources, test the numbers, look for the missing side, and know when to stop and ask a person.

    Elise Moreau