A score that measures memory, not capability.
Measuring a system on the cases that served to build it does not give an optimistic figure. It gives a figure that ranks the options in the wrong order, and leads to buying the wrong thing.
The trap closes at the most natural moment
You build a system from a set of example cases. You use them to write the rules, choose the wording, tune the matching. Then comes the moment to measure, and you measure on those same cases, because they are the only ones whose right answer you know.
The figure obtained has a name in statistics: it is a resubstitution score. It does not measure the system’s ability to handle a case, it measures its fidelity to itself. And it is almost always flattering, because the system was shaped on those precise cases.
The problem is not that it is optimistic. It is that it is optimistic unevenly, and that it therefore ranks the options in the wrong order.
What the measurement gave
We compared three language models run locally, of increasing size, with strictly identical code. Two sets of cases: the one that had served for the construction, and a set of real, unseen cases.
| Local model | Construction cases | Unseen cases |
|---|---|---|
| The smallest, around 2 GB | 67% | 15% |
| Intermediate, around 4.4 GB | 78% | 18% |
| The largest, around 7.1 GB | 89% | 20% |
Twenty-two points of spread on the construction corpus, five points on real cases. The small set ranked the models neatly by size, which is exactly the ranking one expects to find, and that ranking was wrong. On one of the tasks measured, the intermediate model even beat the largest.
The practical conclusion is brutal: on the construction corpus, a bigger model recites better what it has already been given. That is memory, not capability. Any purchasing decision based on the left-hand side of that table would have been a decision taken on noise.
The signal worth more than the score itself
A second finding, from the same campaign, proved more useful still. Two mechanisms with nothing in common were measured on the same cases: on one side a deterministic rules engine, on the other a language model. They plateaued at the same place, within a few tenths of a point of each other.
This signal is easy to obtain and rarely looked for. Having a second mechanism, however rudimentary, is enough to tell whether the observed ceiling belongs to the system or to the data. Once the question is framed in those terms, the work changes in nature: you stop tuning instructions and go looking for what is missing in the material.
How to guard against it, in practice
Four steps are enough, and they are cheap compared with what they avoid.
Separate the sets before writing a single rule. Set aside a share of the cases from day one, never open them during the construction, and accept discovering them late. A validation set put together after the fact, by picking from what is already known, is worthless.
Fix the random seed, so that two successive measurements are comparable. Without it, the gap between two versions blends into the variation from one draw to the next, and an improvement gets the credit for what is merely chance.
Break the results down by pattern rather than following a global score. A single figure can stay stable while one category collapses and another improves. The detail by type of case shows the regressions that an average hides.
Write the warning into the measuring tool itself. This is the most effective of the four. Our evaluation harness states explicitly, next to any figure coming from the construction corpus, that it is a resubstitution score. A spoken convention gets lost between two meetings; a note printed on every run does not.
What this means when you are shown a figure
If a supplier quotes you a success rate, the first question is not its value. It is: on which cases, and did those cases serve to build the system.
An embarrassed answer to that question says more than the percentage itself. And an honest answer, of the kind “here is the score on the construction cases and here is the one on unseen cases, the gap is this much”, indicates a supplier who has done the work, whatever the gap announced.
We apply this rule to our own measurements, including when the gap does not flatter us. It is the only way to make a figure usable by someone who has to decide.