Your AI Can't Count Your Notes — Why 'How Many' and 'List All' Need a Script

Ask a chatbot over your notes how many mention a word, or to list every note about a topic, and it answers from a handful of retrieved snippets — not the whole vault. So the count is a guess capped by what it fetched. For an exact number, run a one-line search over your files.

This failure has a name in the research. Barnett and colleagues, in Seven Failure Points When Engineering a Retrieval Augmented Generation System (CAIN 2024), catalog the ways a retrieval pipeline drops answers "from three case studies from separate domains: research, education, and biomedical."¹

The root cause traces to the original design. In Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020), Lewis and colleagues build the architecture on a retriever that returns "(top-K truncated) distributions over text passages given a query," and in the RAG-Sequence formulation of their §2.1, "the top K documents are retrieved using the retriever, and the generator produces the output sequence probability for each document."² K is a fixed number. Counting and listing ask about all of your notes. Those two facts do not fit together, and no prompt phrasing reconciles them.

Why does my AI get the wrong count when I ask how many notes mention X?

Because a chatbot over your notes reads only the top few chunks a retriever pulls back — typically a fixed number, K — not the full vault. If the word appears in more places than K, the extras never reach the model. It reports what it saw, confidently, and undercounts the rest.

Picture the concrete case. You have forty notes that mention Docker across two years of tinkering. You ask, "how many of my notes mention Docker?" The retriever fetches the eight chunks it scores most relevant, the model reads those eight, and it answers "around eight" — or worse, rounds to a confident "about a dozen." It is not lying. It answered honestly about the slice it was shown. The other thirty-two notes were never in the room.

The aggregation-benchmark paper Aggregation Queries over Unstructured Text (AGGBench, an arXiv preprint, February 2026) states the mechanism plainly: "The relevant mentions often exceed k, and omissions in retrieval propagate irreparably into the reasoning stage, leading to systematic undercounting or incomplete aggregation."³ Barnett's team names the same shape as a distinct failure mode — documents that "did not make it into the context for generating an answer," which happens "when many documents are returned from the database and a consolidation process takes place."¹ The number is wrong before the model does any reasoning.

Does retrieving more chunks (raising top-k) fix the counting problem?

No. Raising K widens the window but changes the failure, not the outcome. Past the point where the relevant chunks fit, the extra text is mostly noise that crowds the prompt and dilutes the answer. Aggregation is a "find all" task, and a retrieval window built to "find one" stays the wrong tool.

The AGGBench authors are direct about the trade: when the window is smaller than the true number of matches, "the resulting upper bound […] can fall far below the true total. Otherwise, increasing k indiscriminately introduces large amounts of irrelevant text, inflates context length."³ You lose either way — undercount with a small K, or drown the signal with a large one.

Their framing of the deeper problem is that "aggregate queries require exhaustive evidence collection and systems are required to 'find all,' not merely 'find one.'"³ A fixed top-k window, they conclude, "makes them fundamentally incompatible with aggregation."³

There is a subtle version of this that catches careful people. Raising K does improve recall — the model does start seeing more of the relevant notes. But recall is not the same as an accurate count.

A model handed sixty chunks, forty relevant and twenty off-topic, does not reliably return "forty"; the off-topic text pulls the estimate around, and the tally wobbles run to run. Barnett's team catalogs this failure even when the evidence is present — answers that "are not incorrect but miss some of the information even though that information was in the context and available for extraction."¹ Better coverage does not translate into a trustworthy total.

The one move: reach for a script, not the chatbot

For any question shaped like "how many" or "list all," run a search over the files instead of asking the model. A one-line command reads every note and returns an exact, repeatable number. Keep the chatbot for what it is good at: reasoning over a small set you already pulled, or summarizing a passage in front of it.

The division of labor is the whole trick. A retriever plus a model is a machine for finding one relevant thing and reasoning about it — genuinely useful, and not what a census needs. A search tool reads all your notes and tallies matches with no judgment and no omission.

This is the capability case, not the speed case; for the cost-and-latency argument that regex often beats a full pipeline, see Don't Build a RAG Over Your Vault. Grep It. The point here is narrower and harder: even with a perfect pipeline, a top-k chat cannot answer "how many" across everything.

Notice which of the two questions each tool is built for. "What does this note argue, and does it contradict that one?" is a reasoning question over a small set — hand it to the model. "Which notes argue this, and how many are there?" is an enumeration question over the whole vault — hand it to a search.

Mixing them up is the entire source of the wrong number. The moment a request contains every, all, how many, or which ones, that is the signal to stop typing into the chat box and open a terminal instead.

Count how many notes mention a word or tag

Three commands cover almost every count. rg -c totals matches, grep -rl with wc -l counts files that contain a term, and find with wc -l counts files themselves. Each reads your notes directly and returns a number you can re-run tomorrow and trust.

Here is the five-minute version, in order:

1. Open a terminal in your notes folder.
2. To count files that mention a word: grep -rl "docker" . | wc -l.
3. To count total mentions (a word can appear many times in one note): rg -c "docker" . | awk -F: '{s+=$2} END{print s}'.
4. To count files carrying a tag: grep -rl "#reading" . | wc -l.
5. To count notes total: find . -name "*.md" | wc -l.
6. Re-run any of them whenever you want the current number.

These are not estimates. On the folder that holds the research for this very post, the exact figures came back instantly with ripgrep 14.1.1 on 2026-07-23:⁴

$ rg -l -i "\brag\b" content -g dossier.md | wc -l      # 47  notes mentioning RAG
$ rg -c -i "retrieval" content -g dossier.md | wc -l     # 44  notes containing "retrieval"
$ rg -o -i "retrieval" content -g dossier.md | wc -l     # 344 total occurrences
$ find content -name "dossier.md" | wc -l                # 172 dossiers total

Forty-seven, forty-four, three hundred forty-four, one hundred seventy-two. A chatbot over the same files would have offered a plausible, round, wrong number — and no way to check it.

Note the difference between the second and third numbers, because it is the one distinction worth getting right. Forty-four notes contain the word "retrieval," but the word appears three hundred forty-four times in total — many notes use it more than once.

"How many notes mention it" and "how many times is it mentioned" are different questions with different commands: grep -rl counts files, rg -o counts occurrences. A chatbot will happily blur the two into a single vague figure; the shell keeps them apart, which is usually what you actually wanted to know.

List every note about a topic

A count tells you how many; a list tells you which. To enumerate every matching note, print the filenames instead of tallying them, then save the command as a tiny script you reuse. The output is the ground truth a chatbot can only approximate — and it stays correct as your vault grows, with no re-indexing.

The list is one flag away from the count — drop the wc -l and keep the file list:

$ grep -rl "docker" . --include="*.md"
notes/infra/docker-compose.md
notes/2024/homelab-setup.md
notes/reading/container-security.md

When you find yourself running it often, save it as notecount:

#!/bin/sh
# notecount TERM [DIR] — list and count notes containing TERM
term="$1"; dir="${2:-.}"
grep -rl "$term" "$dir" --include="*.md"
echo "---"
grep -rl "$term" "$dir" --include="*.md" | wc -l

Now notecount docker lists every file and prints the total. The same file works next month, next year, on ten notes or ten thousand, offline, with the result identical each time you run it against the same notes.

The list also composes in ways a chat answer never will. Pipe it into another search to narrow "notes about Docker" down to "notes about Docker that also mention security" — grep -rl "docker" . --include="*.md" | xargs grep -l "security". Each step is exact, and each step is inspectable: you can see precisely which files survived the filter. That is the difference between an answer you can audit and one you have to take on faith.

Common mistakes

Most miscounts come from asking the wrong tool the wrong way. The fixes are small. Do not trust a round, confident number from a chatbot for a global count; do not paste your whole vault into a prompt to force coverage; and do not confuse a literal match with a meaning-based one.

• Trusting the confident number. A model will answer "about 40 notes" with the same tone whether it read 8 chunks or 800. Confidence is not coverage. Verify any global count with a search.
• Stuffing the context window instead of searching. Pasting every note in to "make sure it sees everything" fights a budget you will lose, and long inputs degrade rather than fix aggregation. That budget has its own trap — see Token Cost Is the New Page Count for Your Notes.
• Confusing counting with computing. Summing the numbers inside one note is a different failure than enumerating notes across the vault; a model pattern-matches arithmetic just as unreliably, covered in Don't Trust the AI to Add Up Your Notes.
• Assuming the index is complete. Before you count, confirm the tool can even see every file. A coverage gap is upstream of an aggregation gap — Ask Your AI What It Can't Read.

Where the chatbot still wins — and the honest scope

This is a limit of one setup: a chatbot answering from a fixed set of retrieved snippets. It is not a claim that models cannot count at all. Give a model a tool — a code interpreter, an agent that can run a query — and it can count, because now it runs the search too, instead of eyeballing chunks.

That is worth stating carefully, because the distinction is the whole lesson. Plain chat-over-your-notes reasons over what a retriever hands it, so a global tally is out of reach by construction. An agentic system that writes and runs grep is doing exactly what this post recommends — it just hides the script behind a chat box. The AGGBench authors build their own method around structured, exhaustive collection rather than raw chunk retrieval, precisely because the window cannot be enlarged into completeness.³

So the rule is not "never ask an AI to count." The rule is narrower and more useful: if the answer must be exhaustive, something has to read every file — a script you run, or a tool the model runs on your behalf. A chat that only reads retrieved snippets is not that something. When you know which kind of system you are talking to, you know which answers to trust and which to check.

How this works with plain-text notes

The whole approach depends on one thing: your notes being plain files you can read directly. When each note is a Markdown file stored on your own device, a search reads every one of them directly, with no index to build and no server round-trip. The exact answer is one command away, offline, and it is yours to re-run.

That is the quiet advantage of keeping notes as open text rather than rows in someone's database. grep, rg, find, and wc are decades old, ubiquitous, and need no pipeline at all; they will read a Markdown file the same way in twenty years.

A note locked inside a proprietary store can only be counted the way that store lets you count it — which is usually through the very chat box that miscounts. Plain files answer to any tool you point at them. If you want the same ownership over your own writing statistics, Count Your Word Count From Your Own Files applies the same habit to a different question. The file is what lasts; the tool is disposable.

Frequently Asked Questions

These are the questions people actually type after a note-AI hands them a count that feels off. The short version runs through all of them: a chatbot answers from retrieved snippets, so trust it for reasoning over a passage and reach for a search whenever the answer has to cover every file.

Can RAG answer "how many" or "list all" questions about my documents?

Not reliably, when it is plain retrieval-and-chat. A top-k retriever passes a fixed number of chunks to the model, so any question needing every match is capped at what those chunks hold. AGGBench calls fixed top-k retrieval "fundamentally incompatible with aggregation."³ Use a search or a tool-running agent for exhaustive answers.

Why does ChatGPT miscount my documents?

Because it answers from the excerpts a retriever selected, not from every file. If more documents match than the retrieval window holds, the surplus never reaches the model, and Barnett's team documents exactly this: relevant documents that "did not make it into the context."¹ The model then reasons over a partial set and reports a partial, confident count.

Should I just use a bigger context window or a long-context model?

A larger window helps only until the relevant text stops fitting, and it introduces a new problem. AGGBench notes that "increasing k indiscriminately introduces large amounts of irrelevant text, inflates context length."³ More text is more noise, not more certainty. For a guaranteed-complete count, read the files with a search rather than betting on window size.

How do I count how many notes mention a word or tag?

Run one command in your notes folder. grep -rl "term" . | wc -l counts files containing the term; rg -c "term" . reports per-file match counts; find . -name "*.md" | wc -l counts notes total.⁴ Each is exact, instant, and repeatable — the number does not drift between runs.

How do I list every note about a topic reliably?

Print filenames instead of counting them: grep -rl "topic" . --include="*.md" lists every matching note. Save it as a small script and it works unchanged as your vault grows.⁵ Unlike a chatbot's list, it never silently omits a file that a retriever failed to surface, because it reads all of them.

Is this the same as the AI getting arithmetic wrong inside a note?

No, and the distinction matters. That is a compute limit — a model pattern-matching the sum of numbers within a single note. This is a scope limit — enumeration across your whole vault, capped by the retrieval window. Different failure, same fix: reach for a deterministic tool instead of the chatbot.

Your notes already know the answer. The question is only whether you ask the files or ask a model to guess at them.

If you keep your notes as plain Markdown files you can read directly, the exact count is always one command away — mnmnote.com.