4 min read

Long documents, short answers: why retrieval gets worse after page 40

We tested how answer quality changes as documents get longer. Accuracy held up to about forty pages, then fell. Here is what caused it and how we fixed it in Library.

Hector SalinasCo-founder and CTO

Research
On this page

Most questions people ask Pilot are about a single long document. A 90-page tender, a 60-page research report, a master services agreement with eleven schedules. Early this year we noticed that answers on these documents were less reliable than answers on short ones, even when the information was clearly there.

This post explains how we measured the problem, what caused it, and what changed in the new Library index that shipped in release 4.2.

How we tested it

We built a set of 1,200 questions across 300 real documents that customers had shared with us for evaluation, with permission and with personal data removed. Each question had a single correct answer that a reviewer could point to on a specific page.

We grouped the documents by length and asked the same model the same questions, using the retrieval system we had at the time. A reviewer then marked each answer as correct, partly correct or wrong. Elise’s team describes the review process in more detail in How we measure whether Pilot’s answers are right.

What we found

Accuracy was stable for short and medium documents, then dropped sharply.

Document length Correct Partly correct Wrong
Under 10 pages 94% 4% 2%
10 to 40 pages 92% 5% 3%
40 to 100 pages 81% 11% 8%
Over 100 pages 72% 15% 13%

The model was not the problem. When we gave it the right passage directly, it answered correctly over 95% of the time at every length. The errors came from retrieval: the system was handing the model the wrong parts of the document.

Why long documents break retrieval

We used to split every document into chunks of roughly 400 words, regardless of its structure. That works for blog posts and short memos. It works badly for long, formal documents, for three reasons.

Definitions live far from where they’re used. A contract defines “Service Credits” on page 4 and applies them on page 61. A chunk from page 61 makes no sense without page 4, and the old index had no way to connect them.

Tables get cut in half. A pricing table that spans a page break was often split across two chunks, so the model saw the column headers in one and the numbers in another.

Similar sections compete. A tender with twelve lots has twelve sections that look almost identical. The old index often returned the right kind of section from the wrong lot.

What we changed

The new index treats a document as a structure, not a stream of words.

  • Sections, not fixed chunks. We split on headings, numbered clauses and page structure, so a chunk is a complete thought. Long sections are split further, but never mid-sentence or mid-table.
  • Context travels with each chunk. Every chunk carries its heading path, such as Schedule 3 › Service levels › Credits, and links to the definitions it uses.
  • Tables stay whole. A table is stored as one unit with its headers, however many pages it spans.
  • Two-step search. We first find the most relevant sections, then search within them, so a question about Lot 7 stays inside Lot 7.

Results after the change

We ran the same 1,200 questions against the new index.

Document length Before After
40 to 100 pages 81% correct 92% correct
Over 100 pages 72% correct 89% correct

Short documents improved slightly too, mostly because tables now survive intact. And because the new index searches fewer, better chunks, it is also faster. Median search time on a 50,000-document workspace fell from 1.9 seconds to 0.6.

What’s still hard

Scanned documents with poor text recognition remain the weakest case, especially older contracts and handwritten annotations. Questions that need information from many places at once, such as list every obligation that applies to the supplier, are also harder than questions with a single answer.

We’re working on both. In the meantime, if an answer from a long document matters, open the citation and check the page. It takes seconds, and it’s the habit we’d recommend even if retrieval were perfect.

libraryretrievalevaluation
Portrait of Hector Salinas

Written by

Hector Salinas

Co-founder and CTO

Built search infrastructure at two companies before deciding answers need sources.

All posts
  • Research

    4 min read

    How we measure whether Pilot's answers are right

    Every week, reviewers grade thousands of Pilot answers against their sources. How the evaluation works, what we count as an error, and the numbers from our last quarter.

    Elise Moreau and Hector Salinas

  • Product

    4 min read

    Release 4.2: Autopilot agents

    Autopilot agents now write your recurring reports on a schedule, from the sources you choose, and wait for a person to approve them. Library search is also three times faster.

    Tomás Ferreira and Hector Salinas

  • Guides

    6 min read

    How to check an AI answer in under a minute

    A five-step routine for checking an AI answer before you use it: read the sources, test the numbers, look for the missing side, and know when to stop and ask a person.

    Elise Moreau