A knowledge base in text files.
What makes a corpus usable by an AI is neither its volume nor the software that hosts it. It is that it should be in plain text, cut into units that stand on their own, and explicitly linked. Measured on ours.
What Obsidian really is
Obsidian is not a note-taking application that stores your data somewhere. It is a folder of text files on your disk, which the software merely displays. Each note is a file in Markdown format, readable in any editor, versionable, copyable, and backed up like any other document.
The property sounds trivial and it changes everything. There is no proprietary database to export, no programming interface to request, no query limit, no format to reverse-engineer on the day the publisher changes its business model. The day you want to do something else with your notes, they are already in the one format that everything can read.
Why it is the right substrate for an AI
A system that has to answer from your documents always starts with the same thankless work: fetching them, converting them, extracting the text, guessing their structure. On a corporate collaboration space, this stage takes up most of the project, and it produces a copy that falls out of step the very next day.
With text files, this stage does not exist. The content is already text, the structure is already marked by the headings, and the update is the file itself. A retrieval system reads the folder, nothing else.
Version control comes as a bonus. A folder of text files can be placed under version management, which gives two rare things: the history of who wrote what and when, and above all the ability to process only what has changed. On a living corpus, reprocessing only the differences rather than the whole is the difference between a daily run and a monthly one.
The graph is laid down by hand, not guessed
This is the most underestimated property. In these tools, one note cites another through an explicit link. On a properly kept working vault, the density we observe is of the order of five links per note, with close to nine notes out of ten linked to at least one other.
These links are information that nobody has to reconstruct. A conventional retrieval system brings documents together by statistical resemblance, which works badly as soon as two texts talk about the same thing in different words, or two texts use the same words for different things. A link laid down by the person who wrote the note says something that resemblance will never say: that these two notes are related, and how.
In practice this allows a search in two stages: find the most relevant note, then follow its links to bring back its neighbourhood. The result is more accurate and considerably smaller than a similarity search that casts a wide net so as to miss nothing.
The real cost is not storage, it is context
This is the question that decides whether a project is feasible, and it is almost always asked back to front. Storing notes costs nothing. What costs is what is sent to the model with every question.
| What is sent in order to answer one question | Tokens |
|---|---|
| One well-segmented note, median value | ≈ 1,200 |
| Three notes targeted by retrieval | ≈ 3,600 |
| The entire corpus, on a real vault | ×71 |
Seventy-one times. And the important point is not the expense: it is that the expensive version is also the least reliable. A model’s ability to exploit a piece of information degrades according to where that information sits in a loaded context, and what lies in the middle of a very long set is used markedly less well than what lies at either end. Sending everything costs more and answers worse.
That is why the sales argument of the very large context window should be received with suspicion. It sells a characteristic of the model supplier as though it were a property of the system, whereas the real question is the quality of the selection upstream. We set this out in detail in our note on bounded AI.
The format has a price, at identical content
Markup counts too. We took one of our published pages and counted the same thing three times: the page as it is served, the article alone with its tags, and the bare text.
| The same content | Tokens |
|---|---|
| The complete page, as served | 4,388 |
| The article alone, tags included | 2,644 |
| The bare text | 2,012 |
The whole page costs more than twice the text it carries. That overhead is markup, navigation and metadata, which is to say noise for a model. On an office document or an export from a collaboration space, the proportion is often worse, and nobody looks at it.
Markdown sits at the other end: it keeps the useful structure, headings, lists, emphasis, with markup reduced to a few characters. It is the format that carries the most meaning per token.
What this requires of whoever writes
A well-kept vault does not happen by decree. Three disciplines are enough, and they are the ones that make retrieval effective afterwards.
A note deals with one subject and one only. That is what makes it possible to bring it back whole without dragging in irrelevant material, and what avoids having to cut it up artificially later. The title announces what the note contains, not the category it belongs to: it is often the title that decides whether a note will be found. And links are laid down while writing, not in a tidying session that never takes place.
A fourth practice makes more difference than the other three together: keeping an index note that gives, in one line, the content of each of the others. It is what makes the first sift possible without reading anything else, at a cost of a few hundred tokens.
What this choice does not do
A method that displays no limits has not been tested. This one has four drawbacks, and they count.
A folder of files is not a multi-user system. There are no fine-grained permissions per document, no comfortable simultaneous editing, no audit trail in the sense of a document management system. For a corpus that has to be shared across an entire organisation with differentiated permissions, it is not the right tool, and a different architecture has to be accepted.
The quality of the graph depends entirely on the discipline of those who write. A density of five links per note is not a property of the software, it is a habit that is kept up. A neglected vault offers none of the advantages described above.
Markdown renders complex tables and documents with constrained layout badly. An agreement, a contract or a formatted report remain office documents, and have to be treated as such. Finally, files on a workstation are only backed up if someone backs them up, which is obvious until the day the disk stops.
These limits mark out a use rather than disqualifying it: a working knowledge base, kept by a small team, meant to be queried. That is exactly the case where the difference in consumption measured above decides whether a project is feasible or not.
On the degradation in the use of a piece of information according to its position in a loaded context, see the work published under the name “lost in the middle” (Liang et al.).