A trust contract for generated reports

1. The problem

A generated report reads confident everywhere. The prose has one register, so a claim backed by a crawled page and a claim the model guessed at look identical on the screen. A reader who has been burned once discards the whole document, and for a report that ends in a Continue, Pause, or Kill verdict on someone’s product, one wrong unmarked claim is enough to lose them.

The fix we shipped is a contract: every claim in the report carries one of three labels, Verified, Likely, or Unknown, and the label is computed in code from where the claim came from. The model is never asked to rate its own output. Self-assessed confidence fails in the same direction as the text it grades, which makes it decoration.

2. The label is a function of provenance

Each claim leaves synthesis with three fields: how it was produced (observed on a source, inferred, or estimated), the analyzer’s confidence, and whether a supporting source survived normalization. The label derives from those fields after generation, in one place, with a test over it.

function deriveClaimLabel(claim: Claim): TrustLabel {
  if (claim.type === "observed"
      && claim.confidence === "high"
      && claim.hasSupport) return "Verified"
  if ((claim.type === "inferred" || claim.type === "estimated")
      && claim.confidence !== "low") return "Likely"
  return "Unknown"
}

3. Quotes checked by substring

Labels rank claims. A second check pins them down: a claim that cites a source must carry a verbatim span from that source, and after generation a function asserts the span exists in the document with a substring test. The test is deliberately primitive. A hallucination check built on another model call can hallucinate; a substring cannot.

The check earned its place early. In one evaluation run a model produced a fluent, plausible answer with zero tool calls, answering from memory with nothing retrieved at all. Every span check failed at once, and the transcript showed a report that would have read perfectly well and sourced nothing. Plausible output, wrong process, caught in code.

4. The labels audited their own system

The contract paid for itself in a way we did not design. Across an evaluation batch, 13 of 15 successful transcripts produced exactly the same counts: two Verified, nine Likely. Real analysis has variance, and identical counts meant the claims were structural. The trail led to component analyzers emitting a fixed two-claim block with hardcoded type and confidence, so the labels were describing the code, and the code was the same every run.

The counter case confirmed the labels track evidence. One product site crawled thin, and its report came out zero Verified, four Likely, eleven Unknown: the label distribution collapsed toward Unknown exactly when the evidence did. The fix that followed let each analyzer emit a variable-length claim list with type, confidence, and support derived from what was actually observed on the page.

5. What to take away

• 1

  Trust labels belong in code, derived from provenance. A model rating its own output fails with the output.

• 2

  A substring is a verification primitive that cannot hallucinate. Use it before anything cleverer.

• 3

  Distributions of labels are a diagnostic. Identical counts across runs means the pipeline is asserting, and measuring it is what showed us.