Per-claim confidence derived from where the evidence came from, and an evaluation split that runs on every commit.
A generated report reads with uniform confidence. The sentence drawn from the page in front of it and the sentence the model inferred from nothing look identical, sound identical, and carry the same authority. For a product whose entire value is a verdict someone will act on, that is the defect worth engineering against.
The system takes public signals about a product, a landing page, pricing, public APIs, distribution footprint, and returns a structured diagnosis rather than prose: a verdict, a set of claims, and a recommended next step. The synthesis is a language model. The problem is that a model asked to assess evidence will produce an assessment whether or not the evidence was there.
Two decisions follow from that, and they are the substance of this report. Confidence is computed rather than asked for, and the evaluation surface is split along the line between what is deterministic and what is not.
The obvious approach is to ask the model how confident it is. That fails in a specific way: self-reported confidence is generated text like everything else, and it correlates with how fluent the claim sounds rather than with whether the claim was observed.
Instead every claim is labeled from the reliability of the source that produced it, in code, after generation. A claim traceable to rendered page content is treated differently from one resting on a public API, which is treated differently again from one the model supplied with no source behind it. The label is derived from the source tier, not negotiated with the model.
Two properties matter. The labeling is per claim, not per report, because a single report routinely mixes observed fact with inference and a report-level score would average the two into something meaningless. And the derivation is a pure function, which is what makes the next section possible.
Evaluating a system with a model inside it is usually treated as one problem. It is two, and they have opposite economics.
The synthesis layer can only be judged against live model calls. A full regression across the reference URLs costs real money and ten or more minutes of wall time, so running it on every commit is not affordable. It runs nightly and before releases.
The contract layer is everything around the model: schema parsing, the label derivation, the usage gate, URL handling, the publish rules. All of it is deterministic. Given a recorded model output as a fixture, every assertion is a pure function call. No network, no keys, no charges, no flakiness. That suite runs in under a second and belongs on every commit.
| Layer | Cost per run | Cadence | Catches |
|---|---|---|---|
| Contract | Zero, no network or keys | Every commit | Schema drift, label collapse, gate bypass, URL routing |
| Synthesis | Real spend, 10+ minutes | Nightly and pre-release | Prompt and model regressions in output quality |
The split is not a convenience. Four consecutive releases added contract surfaces with no automated regression behind them, and none of the failures they could produce would have been caught by the expensive suite, because the expensive suite never exercises those paths in isolation. A schema change can silently break a rendered label. A change to the usage counter can let an unbounded number of free analyses through. Neither is a model problem, and neither shows up in a model evaluation.
There is a third property worth naming: the contract suite needs no secrets, so it runs identically on a laptop, in CI, and in a pre-commit hook. The live suite fundamentally cannot.
The same input run repeatedly does not produce the same verdict. Rather than suppress that, a consistency run executes one input N times and reports per-field variance, with an explicit operational threshold: below ten percent variance on the headline verdict, and no more than one level of movement on the graded score. A field that exceeds it is a bug in the prompt or the scoring function, not weather.
A separate check watches the distribution of graded outcomes across recent runs and warns when it drifts. If most analyses collapse onto the lowest grade, the scoring function has become punitive and the product is quietly telling everyone the same thing. That is invisible in any single report and obvious in the aggregate.