A question-answering system whose every claim can be traced back to the passage it came from.
A model asked to answer from a fixed collection will answer anyway when the collection does not contain the answer, fluently and in the same voice it uses when it is right. This project treats that as the engineering problem: not making answers sound authoritative, but making them checkable, and measuring whether the checks hold.
The target reader is an editor deciding whether to trust an answer. That single requirement determines most of the architecture. Every claim needs a citation that resolves to a real passage, every quotation needs to exist in the source text, and when the collection cannot support a question the system needs to say so rather than fill the gap from the model's memory of the world.
The corpus is small enough to sit in memory: 1,330 chunks at 1,024 dimensions is 5.2 MB, and a full scan takes 4 to 6 milliseconds against a network round trip two orders of magnitude larger. No vector database was used, and at this size one would have added an operational dependency without changing a measurable number.
The first evaluation set was hand-written and scored 24 of 24. A set everything passes cannot answer whether a change helped, so a second, harder set was generated: 60 paraphrased questions whose answer passage is known, worded deliberately unlike the source. A third set of 60 further cases was drawn from disjoint chunks and never tuned against, and it is the one quoted above.
Comparisons between configurations use a paired bootstrap, because both configurations answer the same questions and comparing two independent averages throws away that pairing. A difference is reported as real only when the interval excludes zero.
The fusion constant was wrong by inheritance. Reciprocal rank fusion carries a
constant that implementations almost universally set to 60, a value from a 2009 paper that fused
ranked lists over thousands of documents. Over a candidate pool of a few dozen it swamps the ranks
and fusion degenerates into a vote on set membership. The balance point is derivable: a document
ranked first by one retriever ties one ranked k+2 by both, so with a pool of 24 and
k at 60 that tie is unreachable and the ranks stop carrying information.
Hybrid retrieval is kept for a failure mode, not for its average. On aggregate the paired bootstrap between hybrid and dense spans zero on every metric. What justifies the extra machinery is a specific reproducible class: a minor character named twice in the entire corpus is missed completely by dense retrieval at k=8 and ranked first by BM25. Embeddings smear a rare token toward its neighbourhood, and readers search for names.
Lexical search needed alias expansion to stop losing. BM25 treats a nickname and a formal name as unrelated tokens, and fusion discarded the partial bridge dense retrieval had found, so hybrid scored worse than dense on that query class while winning on rare exact names. Query-time alias expansion into the lexical arm at a reduced weight moved one such pair from zero of five to five of five. One alias link was deliberately left out because the expanded term appears in 411 chunks of one work and 2 of the other, so it would have dragged the wrong document into every query.
A cheap form of contextual retrieval made things worse, and the reason generalizes. Prefixing each chunk with a constant per-chapter summary dropped dense strict recall from 83.3% to 71.7%. A constant prefix adds the same vector component to every chunk in a chapter, making them more alike, when the metric depends on telling them apart. The published technique prepends a chunk-specific summary and is discriminating. Metadata is not context.
Reranking shipped off, then on. It was first rejected on development-set evidence where chapter recall was already 96.7%, a ceiling effect. A paired bootstrap on held-out data showed +8.3 percentage points of chapter recall with a 95% interval of 1.7 to 16.7, and it was turned on. It costs roughly $0.0065 and 4.8 seconds per query.
Three checks run after generation, in code, and none of them rewrites the answer. A reader deciding whether to trust it needs to know which sentence to distrust, not a smoother paragraph.
| Check | Question | Method |
|---|---|---|
| Citations | Does every cited id resolve, and was it retrieved this turn? | Index lookup |
| Quotations | Does every quoted span appear in the sources? | String comparison |
| Claims | Does each sentence follow from the cited evidence? | Model judgement |
The two model-based instruments disagree on real cases, and that is the argument for running both. Asked why one character destroyed another's manuscript, the system supplied a motive that is true of the source work and absent from the retrieved passages. A groundedness judge caught it unanimously, zero votes of three; the per-claim audit examined 9 to 16 claims on the same answers and reported none unsupported.
Three runs of identical code scored 49, 49 and 47 of 52. The deterministic half does not move: retrieval 24 of 24 at 0.9479 mean reciprocal rank, grounding 28 of 28, refusal 4 of 4, every run. The variance sits entirely in the model-judged answer cases, 17 to 19 of 22, which is a property of the instrument rather than of the system. Two runs both landed on 49 and disagreed about which three cases failed, sharing only one between them. Reading the total alone would hide that.
The score dropped from 45 or 46 when misquotation was made a failing condition. Three answers carrying quotations verified absent from the sources had been passing. The lower number is against a stricter test.