Duplicate Notes Poison Your AI's Retrieval

When your notes app feeds an AI several near-identical copies of the same note, the machine has to choose one — and it often chooses a stale copy, burns its limited attention re-reading the repetition, and cites the wrong version. The fix is to consolidate near-duplicate notes into a single authoritative note and retire the rest.

The problem is old and measured, if you look one field over. Training a large model on a corpus full of duplicates degrades it. Katherine Lee, Daphne Ippolito and their co-authors found that "over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data," and that one widely used web corpus held "a single 61 word English sentence that is repeated over 60,000 times."¹ Remove the duplicates and the model improves: it learns to "emit memorized text ten times less frequently and require fewer train steps to achieve the same or better accuracy."¹ That is a fact about training, not retrieval, so treat it as an analogy rather than a proof. But the intuition transfers. A corpus that repeats itself wastes the machine reading it — and your vault is that corpus.

The fix in one sentence: consolidate, don't accumulate

The single habit that fixes this is consolidation: when several notes say almost the same thing, merge them into one authoritative note and retire the copies. Fewer, denser notes give the AI one clean place to look instead of five near-identical ones to choose between, and the copies stop competing for its attention.

This is not a fringe worry. A 2026 arXiv preprint by Daria Berdyugina, Anaëlle Cohen and Yohann Rioual opens by naming the exact failure: "Standard Retrieval-Augmented Generation (RAG) chunking methods often create excessive redundancy, increasing storage costs and slowing retrieval."² Retrieval-augmented generation is the general technique that lets an assistant answer from your files rather than from its training. When they cut that redundancy with lightweight filtering, they measured the payoff: "entity-based filtering can reduce vector index size by approximately 25% to 36% while maintaining high retrieval quality close to the baseline."²

A quarter to a third of the index, gone, with no meaningful loss — because most of it was saying the same thing twice.

Why two copies split the AI's attention

Two near-identical notes do not add information — they divide it. When the AI searches your vault, both copies match the query, so retrieval returns one of them almost at random, and whichever it picks eats a slot in the model's limited context that a genuinely different note could have filled. Redundancy in, precision out.

This is how duplicate notes poison your AI's retrieval. The assistant does not get "more" of your knowledge from three copies of your deploy note; it gets one copy and two wasted slots. A 2025 arXiv preprint introducing a method called AdaGReS puts the mechanism plainly: "standard top-k retrieval often returns redundant or near-duplicate chunks that waste token budget and degrade downstream generation."³ Token budget is the machine's working memory for a single answer. Spend it on repetition and there is less left for the thing you actually asked.

The damage compounds because a retriever cannot rescue a generation it has already starved. Devi Prasad Bal and Subhashree Puhan, in a 2026 arXiv preprint benchmarking retrieval strategies, state the stakes without hedging: "A retriever that returns irrelevant or redundant passages actively hurts generation quality, regardless of how capable the LLM is."⁴ A better model does not save you here. The best model in the world, handed three copies of a stale note, answers from a stale note.

The human fix first: merge near-duplicates into one authoritative note

Start with people, not tooling. Find the clusters of notes that say nearly the same thing, pick or write the one you still believe, fold the useful scraps from the others into it, and retire the copies with a pointer to the survivor. This is slower than a code change and far more durable.

The discipline here is atomicity — one idea per note, and one note per idea. Andy Matuschak states the principle: "The underlying principle I'd call the principle of atomicity: put things which belong together in a Zettel, but try to separate concerns from one another."⁵ The failure mode of ignoring it is exactly the retrieval mess you are cleaning up: "If your notes are too broad, you might not notice when you encounter some new idea about one of the notions contained within, and links to that note will be muddied."⁵ Muddied links, and muddied retrieval.

The workflow, in five steps:

1. Gather the cluster. Search for notes with near-identical titles or opening lines — same topic, different day. Similarity search helps, but eyes on the titles catch most of it.
2. Pick the survivor. Choose the note you still believe, or write a fresh one that says the current thing cleanly. This becomes the authoritative copy.
3. Fold in what is unique. Copy the one paragraph the other versions had that the survivor lacks. Duplicates are rarely identical; harvest the differences before you retire them.
4. Retire the copies with a pointer. Do not just delete — leave a stub that points at the survivor, so a link or a habit that reaches for the old note lands on the new one.
5. Re-check on a cadence. Duplication regrows. A ten-minute sweep every month or two keeps the corpus from silting up again.

Here is the shape of the result. The authoritative note names what it absorbed:

---
title: Deploy process
status: authoritative
supersedes:
  - deploy-notes-2024
  - how-we-ship-v2
  - deploy-scratch
---

# Deploy process

The one current, canonical version. Everything that used to live in
three half-finished notes is folded in here.

And each retired copy becomes a one-line forwarding address, not a landmine:

---
title: Deploy notes (2024)
status: superseded
superseded_by: deploy-process
---

Merged into **deploy-process**. Kept only for history.

That superseded_by pointer is doing double duty, and it is worth being precise about which duty. Consolidating copies is about redundancy — how many notes compete. Deciding which of two disagreeing notes is current is a different job, covered in full in Your Vault Contradicts Itself — Date Your Notes So the AI Knows Which One Is Current. Use that discipline for contradiction; use this one for repetition. They compose.

The index fix second: diversity-aware reranking, and what it costs

Once the corpus is clean, you can ask the retrieval layer to stop returning near-duplicates on its own. Diversity-aware reranking — the decades-old Maximal Marginal Relevance method — rewards a result for being different from what was already picked. It is a real tool with a real cost: tuned for diversity, it can measurably lose recall.

The method is not new. Jaime Carbonell and Jade Goldstein defined it in 1998: "The Maximal Marginal Relevance (MMR) criterion strives to reduce redundancy while maintaining query relevance in re-ranking retrieved documents."⁶ It works by scoring each candidate as a "linear combination" of two things — how relevant it is to your query, and how different it is from what the retriever has already chosen — with a single parameter sliding between pure relevance and pure novelty.⁶ It ships today as a built-in option in common retrieval libraries, which is a decent signal that redundancy is a widely felt problem.

But diversity is not free, and the honest version of this post says so. Bal and Puhan's benchmark found that among the strategies they tested, "MMR records the lowest contextual recall (0.775) and answer relevancy (0.658) among retrieval strategies."⁴ Those numbers come from one controlled biomedical benchmark at a single setting, not a universal law — but the direction is the point. A reranker tuned to avoid duplicates can also skip a passage you needed. That is why the machine fix comes second: it manages a symptom the human fix removes.

Common mistakes

The biggest error is believing that merging notes makes the survivor correct. It does not. Consolidation fixes redundancy — how many copies compete — not contradiction, which of two disagreeing notes is true. The rest of the mistakes come from treating a one-time cleanup as a permanent state, or reaching for the index fix before the human one.

• Confusing redundancy with contradiction. Deduplication improves precision and saves budget; it does not make the surviving note accurate. If two notes disagree, dating and superseding them — not merging — is the fix, and it belongs to a separate discipline.
• Deleting instead of superseding. A hard delete breaks the links and habits that pointed at the old note. Leave a stub with a forwarding pointer so nothing lands on a hole.
• Reaching for reranking first. Diversity-aware reranking is a downstream patch with a recall cost.⁴ Clean the corpus first; reach for the reranker only for the duplication you cannot merge away.
• Over-merging into a giant note. The opposite failure is real too. Fold copies of one idea together, but do not weld five ideas into a single sprawling note — Matuschak's muddied-links problem is what you get.⁵
• Treating it as one-and-done. Duplication regrows every time you jot the same thing on a new day. Consolidation is a habit with an eviction policy, not a migration you finish once.

How this works in your own notes

When your notes are plain Markdown files on your own device, the corpus the AI reads is just your folder — nothing more. You can open it, search it, merge two files into one, and delete the copies yourself, because the notes are yours to edit rather than rows locked inside someone else's service. Consolidation is hygiene you control.

That is the quiet advantage of a local-first, plain-text vault: cleaning the corpus is a file operation, not a support ticket. The same folder your assistant searches is the folder you can open in any editor and prune by hand. If you want to own the index that sits on top of those files as well, that argument is made in Own the Vector Index Next to Your Files; and if your vault is small enough that plain search beats a retrieval pipeline entirely, Grep Beats RAG When Your Vault Is Small makes that case. Duplicates hurt every one of these methods — grep included — which is why corpus hygiene comes before any of them.

Frequently Asked Questions

The questions below come from people whose local AI started giving worse answers as their vault grew — the exact opposite of what more notes was supposed to buy them. The short version: near-identical copies compete for a single retrieval slot, consolidation is the durable fix, and the index-side trick helps but is not free.

Why does my AI keep citing the wrong or old copy of my note?

Because several near-identical copies all match your query, and retrieval returns one of them without knowing which you still believe. AdaGReS's authors describe how "standard top-k retrieval often returns redundant or near-duplicate chunks."³ The machine is not choosing the wrong copy on purpose — it is choosing one of many, and the odds are against you until you consolidate them into a single authoritative note.

Do duplicate notes hurt AI search?

Yes, measurably. Redundant passages "waste token budget and degrade downstream generation," per the AdaGReS preprint,³ and Bal and Puhan find that a retriever returning redundant passages "actively hurts generation quality, regardless of how capable the LLM is."⁴ The parallel from model training is just as stark: deduplicating a corpus lets a model "emit memorized text ten times less frequently."¹

How do I find near-duplicate notes in my vault?

Start with the cheap signal: sort by title and skim for the same topic on different days, and read the opening line of anything that looks familiar. Similarity search — grouping notes whose text is close — catches the rest, including copies with different titles. You are looking for clusters that say nearly the same thing, not byte-for-byte identical files.

Should I merge or delete duplicate notes?

Merge first, then retire — do not just delete. Fold the unique scraps from each copy into one authoritative note, then leave the others as short stubs that point at the survivor with a superseded_by line. A hard delete throws away the one paragraph the copy had that the survivor did not, and breaks any link that pointed at it.

What is Maximal Marginal Relevance (MMR)?

It is a reranking method from 1998 that, in Carbonell and Goldstein's words, "strives to reduce redundancy while maintaining query relevance."⁶ It scores each candidate as a linear combination of relevance to your query and difference from what was already picked. It is useful, but tuning it for diversity can cost recall — in one benchmark it recorded the lowest recall of the strategies tested.⁴

Isn't this the same as dating my notes?

No — different axis. Dating and superseding notes resolves contradiction: which of two disagreeing notes is current, covered in Your Vault Contradicts Itself. Consolidation resolves redundancy: too many copies of the same thing splitting retrieval. A note can be a duplicate without being wrong, and wrong without being a duplicate. Use both disciplines.

A vault does not get smarter by holding the same thought five times; it gets slower, and the reader you point at it gets less sure which copy you meant. The work is not capture — it is deciding, in the folder, which one note survives.

Notes in MNMNOTE are plain Markdown files you can open, search, merge, and prune yourself, so consolidating five near-duplicates into one authoritative note is a thing you do, not a thing you request — mnmnote.com.