The threshold must filter the answer, not the display.
Removing an extract from the list of sources does not stop it influencing the text produced. The result is a contaminated answer that looks clean. Where to place the threshold, how to choose it, and why that choice belongs to the business.
The threshold filters the display, or it filters the answer
A system that answers from documents starts by selecting a few of them. That selection comes with a relevance score, and with a threshold below which an extract is judged irrelevant. So far, everyone does the same.
The question that changes everything is where the threshold applies. In many implementations, it filters the list of sources displayed beneath the answer. The extracts that were set aside go on being passed to the model, so they influence the text produced, and the reader sees an answer accompanied by three clean sources without knowing that the reasoning drew on five others, judged irrelevant by the system itself.
The threshold must therefore apply before, to what is passed to the model. An extract below the threshold does not enter the answer because it does not enter the question. It is a change of one line in the code, and a change of kind in use.
And when nothing passes the threshold
The logical consequence embarrasses most projects: if no extract clears the threshold, there is nothing left to pass on. The system must then say that the point is not covered.
This is the hardest behaviour to get accepted, because it looks like a breakdown. A system that always answers seems to work better, especially in a demonstration, where the questions asked fall within the scope by construction.
Yet on a bounded document corpus, the out-of-scope question is not a rare case. It is the most frequent case after a few weeks of use, when users stop asking the questions they were shown and start asking their own.
How a threshold is chosen, since it cannot be guessed
A threshold is not a constant taken from an example found online. It is a trade-off between two opposing errors, and it is measured.
A threshold set too low lets irrelevant material through: the system answers confidently on the strength of extracts that support nothing. A threshold set too high produces the opposite error, more discreet and no less costly: the system declares that it does not cover a point it does cover, and the user concludes that the tool is good for nothing.
The method consists in assembling a set of questions whose answers are known, deliberately including out-of-scope questions, then varying the threshold and recording two rates: the proportion of wrong answers produced anyway, and the proportion of unjustified silences. These two curves cross, and the operating point is chosen according to whichever of the two errors costs more in your business.
The threshold does not replace the adequacy check
A point to watch, because it is easy to believe the problem solved. An extract can clear the threshold, be genuinely relevant to the subject, and not answer the question asked.
Relevance measures closeness of subject. It does not measure that an answer is there. A document perfectly relevant to the theme raised may be no more than a definition or a cross-reference, or may deal with the general case when the question is about an exception. The threshold does not see that difference, and no amount of tuning will make it see it.
A second, separate check is therefore needed, one that compares the content retained against the actual request. We described why it is necessary in our note on the origin of errors: it is the category of failure that no better-performing model reduces.
The three questions to ask
Faced with a retrieval system, three questions are enough to tell whether it was built or merely assembled.
Does the threshold filter what is sent to the model, or only what is displayed beneath the answer. What does the system do when nothing passes the threshold, and can that be demonstrated there and then on a deliberately out-of-scope question. How was the value of the threshold chosen, and on what set of questions.
The third is the most discriminating. A value taken from an example found online, with no test set behind it, means that nobody has measured what the system lets through, nor what it wrongly refuses.