6 min read

How to check an AI answer in under a minute

A five-step routine for checking an AI answer before you use it: read the sources, test the numbers, look for the missing side, and know when to stop and ask a person.

Elise MoreauHead of Research

Guides
On this page

An AI answer is a draft, not a finding. It can be fluent, well organised and wrong in exactly one place, and that place is usually the number you were going to put in a slide.

My team spends its days measuring how often Pilot’s answers are right. The most useful thing we’ve learned is not about models at all. It’s that a short, consistent routine catches almost every serious error, and that most people skip it because they think checking takes too long.

It doesn’t. Here is the routine we use ourselves. With practice it takes under a minute for a typical answer, and it works with any assistant that shows its sources.

Before you start: know what you’re checking

Not every sentence needs the same attention. Before you read an answer closely, sort its claims into three kinds:

  • Facts you’ll repeat. Numbers, dates, names, quotes and anything that will end up in front of a customer or a manager.
  • Summaries of something you gave it. A recap of your own meeting notes or a contract you uploaded.
  • Suggestions. Headline options, a structure for a report, ideas for a campaign.

Suggestions don’t need checking; they need judgement. Summaries need a quick comparison with the original. Facts you’ll repeat need the full routine below. In most answers that’s two or three sentences, not the whole thing.

Step 1: Open the source behind the claim (15 seconds)

Pick the claim that matters most and open the source attached to it. You are not reading the whole source. You are looking for the sentence or the table row the claim came from.

In Pilot, clicking a citation opens the source with the passage highlighted. If your assistant only gives you a link to a homepage, or no link at all, treat the claim as unsourced. That alone is useful information.

Ask one question: does this passage actually say what the answer says it says? The most common failure we measure is not invented sources. It’s a real source stretched a little too far. A study of 200 nurses becomes “nurses”. “Up to 40%” becomes “40%”.

Step 2: Check the numbers twice (10 seconds)

Numbers deserve a second look because they are easy to get almost right. When you find the figure in the source, check three things:

  1. The unit. Percent or percentage points, thousands or millions, monthly or yearly.
  2. The date. A 2021 survey quoted as if it were current is the single most common number error in our tests.
  3. The scope. Which country, which industry, which group of people.

If the answer did any arithmetic, such as a growth rate, a total or an average, redo it roughly in your head. You don’t need the exact result. You need to know it’s in the right range.

Step 3: Read the source’s own caveats (10 seconds)

Good sources hedge. A report will say “based on a small sample” or “results varied widely between regions”. AI answers tend to drop those qualifiers because they make sentences longer.

Scan the paragraph around the highlighted passage for words like preliminary, estimated, self-reported, in some cases or not significant. If you find one and the answer didn’t carry it over, add it back before you use the claim.

Step 4: Look for the missing side (15 seconds)

This is the step most people skip, and it catches the errors that are hardest to see. An answer can be accurate sentence by sentence and still be misleading, because of what it left out.

Ask yourself:

  • Would someone who disagrees with this conclusion point to a different source?
  • Are all the sources from the same organisation, or the same side of an argument?
  • Is there an obvious recent event the answer doesn’t mention?

A quick way to test this is to ask the follow-up directly: What’s the strongest evidence against this? If the assistant finds something credible, you have a more honest answer. If it finds nothing, that’s a reasonable sign the claim is solid.

The question isn’t whether the answer sounds right. It’s whether you could defend each claim to someone who knows the subject better than you do.

Step 5: Decide what to do with it (10 seconds)

By now each important claim falls into one of four buckets:

What you found What to do
Source says exactly this Use it, and keep the citation
Source says something close Rewrite the claim to match the source
Source missing or unrelated Remove the claim, or find a source yourself
Sources disagree Say so in your writing, or ask an expert

Keeping the citation matters more than it seems. When a colleague reads your draft a week later, the source is what saves them from repeating your check.

When a minute isn’t enough

The routine is for everyday answers: a market figure for a deck, a summary of a policy, background before a call. Some work needs more than a minute, and it helps to know which.

Slow down when the claim is legal, medical or financial advice someone will act on, when it will be published under your company’s name, or when it’s about a specific person. In those cases, read the source in full, find a second independent source, and have someone with domain knowledge review the final wording.

Slow down too when every source is something you uploaded yourself. Pilot can tell you what your documents say. It can’t tell you whether your documents are right.

Make it a habit

Checking gets faster the more you do it, because you start to notice the patterns: the stretched scope, the missing date, the lone source. Within a couple of weeks most people on our team can run the five steps without thinking about them.

A few things make it easier:

  • Check the most important claim first. If it fails, the rest of the answer deserves less trust too.
  • Keep answers short. Ask for three claims with sources rather than a long essay you then have to unpick.
  • Ask for the source first. Find the latest figure and quote the passage is easier to check than tell me the figure.
  • Save good sources to Library, so the next answer starts from material you already trust.

Pilot also does part of this for you. Every answer is checked against its own sources before you see it, and claims that don’t match are flagged in amber. That catches a lot, but it doesn’t replace the last pass by a person who knows what the answer is for. That person is you, and now it only takes a minute.

If you want to see how we measure the rest, read How we measure whether Pilot’s answers are right.

fact-checkingcitationsresearch
Portrait of Elise Moreau

Written by

Elise Moreau

Head of Research

Leads the team that measures whether Pilot's answers are actually right.

All posts
  • Product

    4 min read

    Release 4.2: Autopilot agents

    Autopilot agents now write your recurring reports on a schedule, from the sources you choose, and wait for a person to approve them. Library search is also three times faster.

    Tomás Ferreira and Hector Salinas