Retrieval quality is gated, not asserted

It helps to separate three responsibilities that usually get collapsed into one. The first is measurement: the metrics themselves — Recall@k, MRR, nDCG over a labelled query set. This is the easy part; the metrics are classical, well documented, and a few hundred lines of code. The second is measurement integrity: everything that feeds those metrics — where the queries came from, who validated the expected answers, which cohort of queries a given number actually describes. The third is enforcement: what mechanically happens when the number gets worse.

Most teams build the first, assume the second, and skip the third. The result is a dashboard that computes correct arithmetic over questionable inputs and changes nothing when it drops. Each of the three has its own failure modes and its own fixes — and the integrity and enforcement parts, the unglamorous ones, are where a production retrieval system is actually won.

The most common integrity failure looks harmless: a single, growing pool of test queries that serves every purpose at once. It tracks performance over releases, it checks that fixed bugs stay fixed, and it probes corpus coverage — three jobs with three contradictory requirements. A performance benchmark has to stay stable, or this month's number isn't comparable to last month's. A regression set has to grow with every bug you fix. A coverage probe wants volume and breadth, which in practice means generated queries nobody hand-validated. Let all three live in one pool and the aggregate stops meaning anything: machine-generated queries drift into the reported number, fake miss rates appear, real ones hide behind them, and the trend line quietly compares two different test sets.

The fix is separation by purpose, with a growth model and a quality bar per suite. The benchmark is frozen and content-hashed — the hash makes silent edits impossible, so changing the benchmark is a visible, deliberate decision rather than something that happens. The regression suite is append-only: every fixed bug contributes one permanent query, and the suite becomes the system's memory of everything that ever broke. The diagnostic suite is the place for generated volume — broad, cheap, useful for spotting coverage gaps — and it is deliberately excluded from the headline metric, because diagnostic data is not human-validated. That boundary is the actual decision. The suites themselves are just folders; what makes them an instrument is the rule that diagnostic data never leaks into the number you report.

Separation creates a tension, because the honest suite is also the expensive one. Human-validated queries cost exactly the thing that is always scarce: domain experts' time. So the gold set stays small, and the temptation appears immediately — have a model generate a few hundred test queries and measure against those. The risk is real: you are now grading your system against text no user ever wrote, and a synthetic cohort can score very differently from a human one over the same corpus.

There is a disciplined way to use it anyway: validate the synthetic pool against the human gold set before you trust it with anything. Run both over the same system and compare — if the synthetic set reproduces the human set's difficulty within a few points across several metrics, it has earned a role as a statistical surrogate: volume for strata too thin to measure otherwise, breadth the experts never had time to cover. It still never replaces the human set for the verdict. And the comparison keeps paying off later, in how divergence gets read: if the two cohorts drift far apart after a change, the first suspect is the ground truth, not a synthetic triumph. Test data is data — it needs validation like everything else.

An aggregate metric is an average over strata that can move in opposite directions, so any serious evaluation breaks the number down before reacting to it. The less obvious question is which dimension to cut by — there are usually several candidates, and they tell different stories about the same system.

Here is the version of that choice I keep coming back to. Cutting retrieval results by query intent produced a flattering picture: one segment looked strong, and it was tempting to report that cut. Cutting by target page type — the dimension that maps directly onto a chunking strategy, that is, onto something you can actually change — read materially lower. The intent cut describes your users; the page-type cut describes your system. Only one of them tells you what to do next. Choosing the stratification that makes your own number look worse, because it is the actionable one, is what measurement integrity looks like in practice — and it is precisely the choice a demo never has to make.

Everything so far produces an honest number. Enforcement is what turns it into a property of the system. Mechanically it is almost embarrassingly simple: store a baseline run, compare every new evaluation against it, and fail the run — and the CI pipeline behind it — when recall drops beyond a defined threshold. The question "did this change make retrieval worse?" stops being an agenda item and becomes an exit code.

Two properties matter more than where exactly the threshold sits. The gate has to be automatic — part of the pipeline rather than a ritual someone remembers to perform — and it has to be binding: a failed gate blocks the change, it does not file a ticket. The moment a regression is something a human can wave through under deadline pressure, the gate is decoration.

The loop closes with production. Real user feedback, joined back to the exact chunks that produced each answer, flows into the suites: a bad answer becomes a permanent regression query, a recurring gap becomes a set of diagnostic probes. The evaluation layer stops being a phase that happened before launch and becomes the part of the system that keeps the quality claim true after it.

The honest accounting, because none of this is free. Expert validation is the bottleneck and stays one — gold sets grow slowly, and someone senior has to care. A frozen benchmark ages: the corpus drifts, the users drift, and at some point re-freezing is the right call — a versioned, deliberate act that costs trend comparability the day it happens. The gate adds friction to every change, including the innocent ones; that friction is the feature, but it is real, and the evaluation has to be cheap enough to run on every change or people will route around it. And the whole layer is code — a system you maintain next to the system it measures.

Where the data is public and nobody depends on the answers, this is over-engineering; a notebook and a spot check are fine. The calculus flips the moment real users rely on what the system says. Then the evaluation layer is cheaper than the first silent regression that reaches them — you just pay for it earlier, and on purpose.

Stripped to its load-bearing parts, the layer is five decisions and a labelling rule:

None of this requires a platform. It is a few hundred lines of evaluation code, three folders of queries, and a handful of decisions about what counts — most of them made once. The model behind your system will keep improving without your help. Your quality claim will not. It stays true exactly as long as something is built to keep it true — and that something is the gate, not the slide.