Where the errors really come from.
A prototype score does not say what to fix. The useful step is to take the failures one by one and decide, for each of them, whether it comes from the system or from what the system was given to apply. On a real case, the answer redirected the assignment.
A score does not say what to fix
A document prototype always ends up producing a percentage. The system handles a certain proportion of cases correctly, and the team immediately starts discussing the model: should a bigger one be taken, should the instructions be adjusted, should retrieval be enriched.
That discussion is premature, because the score says nothing about where the defect lies. A system can fail because it misunderstood the request, because it brought back the wrong document, or because the expected answer exists nowhere in the reference framework it was given. These three failures look alike in a results table and call for entirely different work.
The only step that separates them is tedious and cannot be automated: take the failures one by one and decide, for each of them, where the cause sits.
Attributing each failure, one by one
On a recent prototype, we went back over every failing case and attributed each one to a single cause. The result redirected the assignment.
| Origin of the failure | Share of failing cases |
|---|---|
| The written reference framework does not cover the situation | 52.1% |
| Our system | 12.5% |
| Other causes, including incomplete input data | the remainder |
More than half the failures did not come from the system. They came from the system being asked to apply a rule that was written down nowhere. No setting, no better-performing model, no improvement in retrieval changes any of that: the right answer does not exist in the corpus.
What the ceiling changes about a project
Announcing this ceiling to a client is uncomfortable, and it is the most useful conversation of the assignment. It moves the project from a technical undertaking towards a doctrinal one: what has to be produced is not a better artificial intelligence, it is the missing rules, written, validated and enforceable.
It also has a practical consequence. Each missing rule becomes an identified line, with the number of cases it would unblock. The list of gaps can therefore be ranked by volume, and the decision to write a given rule or not rests on a figure rather than on an impression.
One check is worth adding, and it matters: an enriched version of the reference framework was supplied to us during the assignment, and we ran it again over the same sample rather than taking it on trust. It filled none of the gaps identified, and its measured effect amounted to a handful of cases. An enrichment that looks substantial on reading may change nothing where it counts.
Deterministic rules first, the model as a fallback
The reflex is to have every case handled by the model. It is the quickest thing to build and the hardest to defend afterwards.
We proceed the other way round. A set of deterministic rules, written in code, first handles what can be decided without interpretation. On this prototype, those rules absorbed more than half the volume with no call to the model at all, at equal accuracy. The model comes in only as a fallback, on what the rules were unable to settle.
Three benefits, in that order of importance. The deterministic part is testable: it can be covered by unit tests, and a regression shows. It can be explained to an auditor, which no model output really can. And it consumes nothing, which takes the question of cost out of the debate.
The corollary is worth stating: the proportion of cases handled without the model is an indicator of the maturity of the reference framework. The more precise it is, the larger the deterministic share grows. A vague corpus pushes everything towards interpretation.
Retrieve by identifier, never by resemblance
An objection was put to us during the assignment, and it was well founded. The setup displayed a match between the case being handled and comparable cases, which gave the impression of a magic button: the system would find the answer by looking for what resembles it.
The answer fits in one sentence, and it is a point of architecture. The answer is retrieved from the reference framework by identifier, deterministically. Resemblance serves only to complete the display, never to decide. The test that proves it is simple: the similarity matching can be removed entirely without changing a comma in the document produced.
Generate nothing that commits
On a document that produces a legal or contractual effect, the rule is absolute: the model does not draft. The reference text is reproduced word for word from the validated corpus, and the model does no more than assemble and personalise what needs to be.
This constraint is easier to hold to than it looks, because it matches what a professional already does: a clause is not reinvented, the validated template is taken and adapted. A system that generates binding content does not do better than a human, it does something nobody has the right to do.
Making the gap visible, not just the failure
The arrangement that made the attribution possible deserves describing, because it transfers. We built a second reference corpus, independent of the client’s, and it is consulted only when the main reference framework stays silent on the situation.
That conditional trigger makes all the difference. If both corpora were queried together, the system would answer better and nobody would see that the client’s reference framework was inadequate: the gap would be filled silently. By calling on the second only when the first is silent, each recourse becomes a signal, countable and attributable on screen.
It is a counter-intuitive choice: we deliberately degrade the apparent performance of the system in order to make a piece of information visible. That information is worth more than the score points it costs.
The error that no better model will reduce
One category of failure resists every improvement, and it is the one we are working on next. The system correctly identifies the category of the case, retrieves the document intended for that category, and yet that document does not deal with the particular situation of the file.
Nothing in the chain is wrong. The classification is right, the retrieval is right, and the result is inadequate. No better-performing model corrects this, because there is no comprehension error to correct: what is missing is a check, one that compares the content of the document selected against the actual content of the request, and refuses rather than delivers when the comparison fails.
This category is why we are wary of stage-by-stage performance tables. A chain in which every link shows a good score can produce an unusable result, and only an end-to-end evaluation, on unseen cases that played no part in the construction, reveals it.
What transfers
Four steps, applicable to any system that applies a written reference framework to particular cases.
Evaluate on unseen cases that played no part in the construction, without which the score measures the system’s memory and not its capability. Attribute each failure to a single cause, which costs hours and redirects weeks. Compute the ceiling at constant reference framework, so as to know how much of the remaining gap is a computing matter and how much is a writing matter. And instrument the silence of the reference framework, so that what is missing shows up instead of being filled in without a sound.
The outcome of this approach is not only a system. It is a list of missing rules, ranked by the number of cases they would unblock, which the client keeps even if it changes supplier.