A system scores 84 out of 100 on Monday. Somebody edits a prompt. On Thursday it scores 88. The question that decides whether the edit ships is not "did the number go up," it is "is a four-point move larger than what this test would produce by doing nothing at all." Usually it is not.
Treat each case as pass or fail and the score is a proportion. Its standard error is:
At p = 0.84 and n = 100, that is about 3.7 points, so a 95% interval
spans roughly 77 to 91. The Thursday score of 88 sits comfortably inside Monday's interval. Nothing
has been demonstrated. To resolve a genuine four-point difference at that baseline you need several
hundred cases, and most eval sets are built by hand and number in the dozens.
This is the single most common error in LLM evaluation work: a suite of 40 cases, a two-case swing, and a decision made on it. Two cases out of 40 is five points of score and well inside the noise of the instrument.
The picture improves considerably if both versions answer the same questions, because then the comparison can be made case by case instead of score against score. Ignore the cases where both versions agree, which carry no information about the difference, and count only the ones where they disagree: A right and B wrong, or the reverse. If a change flips eleven cases in its favor and two against, that is strong evidence from a set of 40, where a five-point score difference was not.
The practical version is a paired bootstrap. Resample the case list with replacement a few thousand times, recompute the difference between the two systems on each resample, and read off the 2.5th and 97.5th percentiles. If that interval contains zero, the change has not been shown to help. It costs a few lines and it answers the question the raw scores cannot.
The reason pairing matters so much here is that eval cases are wildly unequal. Some are easy for every version, some are impossible for all of them, and a handful sit near the boundary and decide the score. Comparing two independent averages throws away the knowledge of which cases those were.
Run the identical system twice and the score moves, because sampling is stochastic and because any model-based grader has variance of its own. That makes a single number misleading in a specific way: it invites comparison against a future single number that carries the same hidden spread.
Running the suite three times and reporting the range is cheap and much more honest. It also tends to surface something the total hides: two runs can produce the same score while failing different cases. If a suite scores 47, 49 and 47, the useful statement is that it scores 47 to 49 and that the variance is concentrated in the graded, open-ended cases while the deterministic ones never move. That sentence tells you where to spend effort. "49" does not.
A perfect score means the set no longer discriminates: it cannot answer "did this change help," because there is no headroom left for a change to show up in. Perfection is a signal to make the set harder, not a milestone.
The usual repair is generated cases. Take known passages, have a model write questions whose answers live in them, and keep the source as the label. That produces a larger and harder set with known ground truth, and it will be measurably harder than the hand-written one, which is the point.
Hold some of it back. A set that every change is tuned against stops being a measurement and becomes a target, and the gap between a tuned score and a held-out score is the part worth reporting.
When a model grades the output, the grader is a second system with its own failure modes, and raw agreement with human labels is a poor way to check it. On a set where 90% of cases pass, a grader that approves everything agrees with the humans 90% of the time while catching none of the failures that matter.
Report the two directions separately: of the answers a human called bad, what fraction did the grader catch, and of the answers a human called good, what fraction did it pass. A grader can be useful with a known bias as long as the bias is known. If the numbers are poor, changing the grading model is usually more effective than rewriting its instructions.
It is also worth running two different checks rather than one. A whole-answer judgement and a per-claim audit disagree on real cases, catching different failures, and the disagreement is information rather than a defect.
Constructing the set is most of the work, and it is not a task that can be delegated to whoever is least busy: someone who knows what a good answer looks like in the domain has to sit with real outputs and write down what is wrong with them. Everything downstream is arithmetic by comparison.
The reason to pay it is that without a measurement, quality is a matter of opinion, and the opinion that wins is whoever spoke last or most recently looked at the demo. With one, a change earns its way in and the argument ends. Nothing else about running a probabilistic system in production is harder to retrofit later.