Your Vault Has Too Many Almost-Right Notes: Why Near-Relevant Clutter Makes the AI's Answer Worse

An AI assistant gives worse answers as your notes grow because the notes that score highest in search are often the ones that look like the answer without containing it. The fix is a curation habit: prune and scope the set the model retrieves over, so near-relevant clutter stops crowding out the one note that actually answers you.

This is a measured effect, not a hunch. In The Power of Noise, a peer-reviewed SIGIR 2024 study, Cuconasu and seven co-authors report a counter-intuitive result: "the retriever's highest-scoring documents that are not directly relevant to the query (e.g., do not contain the answer) negatively impact the effectiveness of the LLM."¹ They give this clutter a name — distracting documents — and a precise definition: notes "semantically similar to the query but do not contain the correct answer."¹ These are not junk. They are your almost-right notes: the meeting recap that mentions the project but not the decision, the older draft that circles a topic without stating its conclusion. They rank near the top of a search because they resemble the answer. That resemblance is exactly what misleads the machine reading your vault.

The single discipline: curate what the AI reads, don't hoard what you own

The habit that fixes this is curation of the retrieved set, not of your whole library. You keep every note you want; you narrow what the assistant is allowed to pull for a given question. Cuconasu's team found a tight window works best: "retrieving between 3 and 5 documents is the most effective choice."¹

Precision beats recall here, and that inverts the instinct most note-takers bring to a growing vault. More notes feels like more to draw on. For a retriever it is more to sift, and every near-miss it hands the model is a slot the real answer could have filled. The authors state it flatly in their conclusions: top-scoring retrieved documents that do not contain the answer, "when added to a prompt, negatively impact the LLM effectiveness."¹

So the move is not to write a better note. It is to keep the almost-right ones out of range.

Why an almost-right note hurts more than a random one

The surprise in the research is an asymmetry: a note that sits close to your question but does not answer it does more damage than a note about something unrelated. Cuconasu's team measured that "adding just one distracting document causes a sharp reduction in accuracy, with peaks of 0.24 (−25%)."¹

And it compounds. As more distractors piled into the context, accuracy kept sliding — deteriorating by more than 0.38, a 67% drop, in some of their cases.¹ One near-miss costs you; a folder of them can sink the answer.

Their illustration is worth keeping. "For instance, if one asks for the color of Napoléon's horse, a passage describing the color of Joséphine de Beauharnais' (Napoléon's first wife) horse, while not containing the right information, would be highly related."¹ It is on-topic, it scores high, and it answers the wrong question. Your vault is full of these — notes adjacent to what you meant.

The same paper reports a stranger result, and it is the one most likely to be misread: "adding random documents in the prompt improves the LLM accuracy by up to 35%."¹ Read that carefully. It is not a licence to hoard. The gain shows up only in an arranged setup where a genuinely relevant note is still present and the random text sits far from the query, and the authors explicitly call for more research into why it happens.

The usable lesson is narrow: unrelated noise is cheap, but near-relevant clutter is expensive, because it scores high and wears the costume of an answer.

A caution on the numbers: these figures come from question-answering benchmarks — Wikipedia questions for Cuconasu's team, grade-school arithmetic for the corroborating study below — not from anyone's personal notes. Treat the mechanism as the transferable part and the exact −25% or −67% as benchmark-specific, not a forecast for your vault.

The effect is not one team's artefact. A field over, Shi and colleagues built an arithmetic benchmark seeded with irrelevant sentences and found "the model performance is dramatically decreased when irrelevant information is included."² Two tasks, two research groups, one direction: text that is present but off-target pulls the answer off with it.

The five-minute version

When an answer comes back wrong or vague, do not reach for a better note — subtract the near-misses. In five minutes you can narrow the set the assistant reads for one question, watch the answer change, and learn which almost-right notes were doing the damage. Precision is a setting you can turn, not a rewrite you owe.

1. Name the question you actually asked, in one plain line. Half of retrieval trouble is a query broader than your intent.
2. Scope the source. Point the assistant at the folder or tag that holds the answer, not the whole vault. A smaller pool has fewer things that merely resemble the answer.
3. Cap the retrieved set. Ask for the top three to five notes, not twenty — the window Cuconasu's team found most effective.¹
4. Read what it pulled. If your tool shows the retrieved notes, skim them. The almost-right ones are obvious the moment you look: on-topic, off-answer.
5. Cut the distractor from range. Move or exclude the note that circles the topic without answering it, then ask again.
6. Compare the two answers and keep the tighter framing that produced the better one.

Concretely: you ask "what did we decide about the retry limit?" and the assistant answers "three," which is wrong. Look at what it read and you find three near-misses ranking above the real note — a planning doc that proposed three, a support thread that mentioned retries, and last quarter's spec. Scope the question to your decisions/ folder, cap it at five notes, and it reads the note recording the actual call: five, capped. Nothing was deleted; the almost-right notes were simply out of range.

The thirty-minute version

The longer version turns a one-off fix into a habit the vault keeps. Instead of scoping a single query, you find the notes that surface as almost-right across many questions and give them a place the retriever can skip by default. Thirty minutes buys a corpus that stays precise as it grows, rather than one that silts up.

Start by keeping a short distractor list for a week: each time the assistant leans on a near-miss, jot the note down. Patterns appear fast. Most repeat offenders share one trait — they are titled by their subject rather than their answer, so they rank for every neighbouring query. Retitle them by the conclusion they carry ("Deploy runs on the blue cluster") instead of the area they touch ("Deploy notes"), and they stop matching questions they cannot answer.

Then handle the fragments. A half-finished draft that circles a decision without stating it is a distractor until it either states the decision or is archived; fold it forward or move it out. Give near-relevant reference material — old drafts, meeting recaps, background reading — its own folder you can exclude from retrieval, so it stays available to you but invisible to the assistant unless you ask.

Last, set a default retrieval cap of three to five notes and widen it only on purpose. Re-check monthly, because near-relevant clutter regrows every time you jot a new note beside an old one.

Common mistakes

Most of these errors come from treating a curation problem as a collection problem — trying to repair a bad answer by adding one more note. The rest come from confusing this failure with its two cousins: too many copies of one note, and two notes that disagree. Each looks similar and each has a different fix.

• Reading "random helps" as "clutter is fine." The +35% result holds only when a relevant note is also present; it is not permission to pad the vault.¹ Near-relevant notes remain the expensive kind, whatever unrelated noise does.
• Confusing precision with correctness. Pruning distractors gives the model a cleaner set to read; it does not make a wrong note right. When two notes disagree on a fact, dating and superseding them is the fix — a separate discipline covered in Date Your Notes So the AI Knows Which One Is Current.
• Confusing distraction with duplication. Near-identical copies of one note are a different failure — redundancy, not distraction — with its own fix in Duplicate Notes Poison Your AI's Retrieval. A distractor is a different note that merely looks relevant, not a copy.
• Expecting a fixed percentage. The −25% and −67% figures come from research benchmarks, not from your notes.¹ Trust the direction, not the number.
• Reaching for a bigger model. A stronger model handed the same almost-right notes answers from the almost-right notes. The leverage is in what you feed it, not how clever it is.

How this works in your own notes

This discipline is only possible when you can see the set the AI reads. When your notes are plain Markdown files on your own device, the corpus is just your folder — you can open it, search it, narrow the assistant to one folder, and delete the almost-right note yourself. A black-box store hides which notes entered an answer.

That visibility is the quiet advantage of a local-first, plain-text vault: you scope retrieval the way you scope a search — by folder, by date, by hand. If your vault is small enough that plain search beats a retrieval pipeline, Grep Beats RAG When Your Vault Is Small makes that case. Either way, watching what the machine pulled is what lets you cut what it should not have read — you cannot cut what you cannot see.

Frequently Asked Questions

These questions come from people whose local assistant got worse as their notes multiplied — the reverse of what more notes was supposed to buy. The short answer: the notes that look most like your question without answering it are the ones crowding out the real one, and the fix is to narrow what the model is allowed to read.

Why does my AI assistant give worse answers when I add more notes?

Because growth adds near-relevant notes faster than answering ones, and the near-relevant ones score high in search. Cuconasu's team found that highest-scoring documents that do not contain the answer "negatively impact the effectiveness of the LLM."¹ More notes means more look-alikes competing for the few retrieval slots your real answer needs.

Does a bigger vault make RAG worse?

Not by size alone, but by what size brings: more notes that are close to a question without answering it. Those distractors are the costly kind, and they accumulate as you write. The paper measured accuracy "deteriorating by more than 0.38 (−67%) in some cases" as distractors increased.¹ Curate the retrieved set and a large vault stays usable.

Why does my RAG return similar-but-wrong documents?

Because retrieval ranks by resemblance, and an almost-right note resembles your query closely — that is its whole problem. Cuconasu's team define these distracting documents as ones "semantically similar to the query but do not contain the correct answer."¹ The retriever is doing its job; it just cannot tell "about your topic" from "answers your question."

Should I prune my notes for better AI retrieval?

Prune what the assistant retrieves, not necessarily what you keep. Scope it to the right folder, cap the pulled set at three to five notes, and exclude near-relevant reference material by default. You get precision without deleting anything you value — the goal is a clean set per question, not a smaller library.

How many documents should I retrieve for RAG?

For most questions over a personal vault, few. Cuconasu's team report that "retrieving between 3 and 5 documents is the most effective choice," because adding more raises the odds of pulling in distracting, counterproductive notes.¹ Start at three to five and widen the window only when an answer genuinely needs more sources.

Isn't this the same as removing duplicate notes?

No — different axis. Duplicates are near-identical copies of one note, a redundancy problem covered in Duplicate Notes Poison Your AI's Retrieval. A distractor is a different note that only looks relevant. A note can be a duplicate without being a distractor, and a distractor without being a copy. Both hurt retrieval; the fixes are not interchangeable.

A vault does not answer better because it knows more; it answers better because it can find the one note that fits and leave out the ten that merely resemble it. The work is not collecting — it is deciding, for each question, what the machine is allowed to read.

In MNMNOTE your notes are plain Markdown files on your own device, so scoping what an assistant reads — one folder, a handful of notes, the almost-right ones left out — is something you do by hand, not something you file a request for: mnmnote.com.