An AI that can say it does not know.
Promising the absence of error does not survive the first counter-example. What can be defended is error that is bounded, traced and measured. Here is the architecture that this requires, and what you can demand of a supplier.
The promise that must never be made
“Our artificial intelligence does not hallucinate.” The sentence sells well, and it does not survive its first real test. One counter-example, a single one, is enough for the client to stop believing everything else: the availability figure, the timetable, the security. An absolute promise turns any minor incident into proof of a lie.
The problem is not commercial, it is technical. A language model always produces an answer. It has no internal mechanism that lets it establish that it does not know: it strings together plausible words, and the plausible is exactly what resembles the true. Promising the absence of error amounts to promising that a tool will behave against its nature.
What can be promised instead
The defensible promise is not about the absence of error. It is about the bounds placed around the error that remains possible, and about the fact that it is measured.
In practice this gives four commitments, all four of which can be checked. The system can assert nothing that is not in your data. Every sentence cites its source. Every figure is computed rather than produced by the model. The fidelity rate is measured continuously on a test set, not announced once in a brochure. And anything falling below the threshold goes to human validation instead of being delivered.
The difference with an absolute promise is that a client can ask to see. A test set can be inspected, a score recomputed, a threshold moved. None of this is a matter of trust.
The architecture that makes the promise true
A doctrine that does not change how the thing is built is worth nothing. This one has precise consequences for the way the system is assembled, and the main one consists in taking away from the language model everything it does badly.
Facts, identifiers and amounts come from your database or your documents, never from the model’s memory. Calculations, scores and criticality rules are written in code, therefore testable, therefore correctable. The model only comes in at the end of the chain, to format what it has been given.
The rest rests on three construction rules. Retrieval comes before generation, which rules out dumping an entire case file into the context and hoping the model will find its way. Instructions are atomic: one task per call, with an output in a strictly defined format, rather than one sprawling instruction that asks for eight things at once. And confidence is measured field by field, not globally, because a document can be read perfectly on nine fields and be illegible on the tenth.
The risks that remain, and they do remain
An honest doctrine names what it does not cover. Four risks remain in this architecture, and we would rather write them down than discover them with you.
Retrieval can bring back the wrong source: the system is then faithful, but faithful to something false. Formatting can betray the content, more rarely, but the risk is not nil even when the model does nothing more than package. The classification of the request can go wrong and route to the wrong procedure. Finally, a piece of data can be reliable and out of date, which no fidelity mechanism detects.
These four risks are handled by the same means: citing sources so that a human can check, setting thresholds that trigger a review, and measuring continuously instead of assuming that what worked last month still works. None of them is handled by an assertion.
A very large context window is a sales argument
One argument comes up in almost every offer: the giant context window, presented as the solution to the reliability problem. It would be enough to give the model everything for it to find what it needs.
The published research says the opposite. A model’s ability to use a piece of information degrades according to where that information sits in a loaded context: what lies in the middle of a very long document is used markedly less well than what lies at either end. Loading more does not make things more reliable, it moves the problem.
An offer that boasts about the size of its window without showing its instruction architecture is selling a characteristic of its model supplier, not a property of its system.
Teaching a system to keep quiet
The most useful work we do on our own products does not consist in improving the answers. It consists in obtaining silences in the right places.
An extract whose relevance falls below a measured threshold does not enter the answer. And above all, it does not enter the material passed to the model either, which is not the same thing: an extract removed only from the display goes on influencing the answer, and the system then appears to be citing sources cleanly when those sources support nothing. If no extract at all passes the threshold, the system says the point is not covered, instead of padding.
This is a costly requirement. A system that always answers makes a better impression in a demonstration, and the gap only shows in production, on the day someone relies on an answer that should never have existed.
What you can demand
If you are assessing an artificial intelligence offer, four requests are enough to separate those who have measured from those who assert.
Ask for the error rate per field, on a test set you can inspect. Ask what the system does when it does not know, and have it demonstrated on a question outside the scope. Ask which figures are computed and which are produced by the model. Finally, ask what risks remain, bearing in mind that an answer along the lines of “none” disqualifies the offer more surely than any admitted limitation.
We apply these four questions to our own systems before proposing them, and we publish the defects we find in them. It is the only way we know of making a promise verifiable rather than credible.
On the degradation in the use of a piece of information according to its position in a loaded context, see the work published under the name “lost in the middle” (Liang et al.).