Retrieval over a closed document collection

1. The system

The brief: a tool for a book publisher’s editorial team over two supplied novels, Little Women and Pride and Prejudice, 307,689 words together. Editors ask grounded questions, get answers with citations that resolve to book and chapter, and look for passages that sit close to each other across authors, which is a rights desk’s question about a submission.

It shipped as a browser application on a Node and TypeScript backend, because the users are editors and rights lawyers. Passages appear verbatim with their citations, and every number in the interface is one a reader can act on. The design log for the project runs to 102 entries, each recording what was chosen, what lost, and what evidence would change it.

2. Two retrieval arms and a fusion

Ingest splits on document structure. The two books mark chapters differently, and a per-book pattern silently breaks on the third book, so the segmenter keys on the shared structural signal instead:

Little Women:       <h2>I. Playing Pilgrims.</h2>
Pride & Prejudice:  <h2><a id="Chapter_I"></a>CHAPTER I.</h2>

3. Checks that run after the model

Retrieval is keyword and dense search fused with reciprocal rank fusion, a reranker over the final candidates, and the retrieval depth chosen from measured context precision on the evaluation set. At this corpus size the index lives in process; a vector database earns its operational weight later, and the decision log records the threshold.

The model runs inside a bounded tool loop with nine tools, and book scope is enforced in the tool layer, so a question scoped to one book cannot read the other. After generation, three verifications run in code: citations must resolve to a real chapter, a per-claim audit separates what the sources support from what they merely fail to contradict, and quotations are checked inside the chapter they claim.

4. The evaluation that found the gap

The evaluation holds 52 cases, grouped by the four jobs the brief names plus a guard set for questions the system must refuse:

5. What the grouping revealed

An earlier version scored by mechanism, retrieval quality here, answer quality there, and the mechanism view hid a hole: the first whole-book question scored zero on groundedness every single run, because eight retrieved passages cannot describe a novel. Grouping the evaluation by the user’s job made the gap visible in one report, and the fix was a missing capability, a collection-level view the mechanism scores had no slot for.

The suite also carries planted defects, cases designed to fail if a specific guard weakens, so a regression announces itself as a red row and a paired comparison decides parameter changes.

6. What to take away

• 1

  Segment on document structure. Per-document patterns are debt that comes due on the third document.

• 2

  Verification belongs after generation, in code: resolve citations, audit claims, check quotes inside their claimed chapter.

• 3

  Group evaluations by the user’s job. Mechanism scores can all be green while a whole job fails.