Ask Your AI What It Can't Read

You cannot trust an AI's answer about your notes until you know how many notes it can see. Count the files on disk yourself, ask the AI to count what it reads, and diff the two numbers. The gap — not the answer — is the thing worth investigating first.

A local AI over your notes is a retrieval system, and retrieval fails quietly. Retrieval-Augmented Generation, as Simon Willison defines it, "is a technique for adding extra 'knowledge' to systems built on LLMs, allowing them to answer questions against custom information not included in their training data." ¹ The knowledge only helps if it arrives.

Anthropic's own tests found a contextual-retrieval method that "can reduce the number of failed retrievals by 49% and, when combined with reranking, by 67%" ² — which is another way of saying that in a plain setup, failed retrievals are common. Coverage failure sits one layer below that: before a chunk can rank badly, the file has to be seen at all. This post is the smoke test for that layer.

Why can't the AI find a note you know is there?

The note is there. The AI just never received it. A local RAG tool or file-reading agent walks your folder through filters — for hidden files, ignore rules, encodings, file types, and permissions — and each filter can drop a file without warning. The answer you get is confident and incomplete, and nothing in it flags the omission.

That silence is the whole problem. A search box that returns nothing tells you it found nothing. An AI assistant asked "what did I decide about the vendor contract?" will answer from whatever files reached it, and phrase the answer with the same confidence whether it read all forty notes or only thirty-one.

Even a well-tuned pipeline leaves gaps: Anthropic measured contextual embeddings alone reducing "the top-20-chunk retrieval failure rate by 35% (5.7% → 3.7%)" ³ — the failure rate drops, but it never reaches zero. Coverage and retrieval are two different failures, and both are invisible from inside the chat window.

The five-command smoke test

The test is a diff. Establish ground truth with one command — count every regular file on disk — then ask the AI to report how many files it can see over the same folder. If the two numbers disagree, the difference is your coverage gap. Five commands take it from a raw count to a named cause.

Run these against a real notes folder. They assume ~/notes; substitute your own path.

1. Establish ground truth. Count every regular file. In POSIX find, the -type f primary evaluates true for a regular file, so this counts files and ignores directories ⁴:

find ~/notes -type f | wc -l

Write the number down. This is the only number in the exercise you fully control.

2. Ask the AI to count what it sees. In the same session you use to query your notes, ask the assistant to enumerate rather than summarize: "List every file you can see under ~/notes, one per line, then give me the total count." Save its list to a file — ~/ai-view.txt — and count it the same way:

wc -l ~/ai-view.txt

3. Diff the two counts, then the two lists. If the totals already disagree, you have your answer. To see which files are missing, sort both and compare. comm -23 prints the lines in the first file that are absent from the second:

find ~/notes -type f | sort > ~/disk-view.txt
sort ~/ai-view.txt -o ~/ai-view.txt
comm -23 ~/disk-view.txt ~/ai-view.txt

Every path comm prints is a file on disk the AI never named.

4. Diff the file types, not just the count. A tool can see a file and still refuse to read it. A histogram of extensions shows what your folder actually contains, so you can compare it to what the AI reports ingesting — often only .md, silently skipping everything else:

find ~/notes -type f | sed 's/.*\.//' | sort | uniq -c | sort -rn

If .pdf, .png, or .docx sit near the top and the AI only quotes Markdown, the gap is a file-type gap.

5. Ask the AI what it could not parse. Most agents and RAG tools log skips. Ask directly: "List every file under ~/notes you could not read or parse, with the reason for each." An honest tool will hand you its own error list — encoding failures, permission denials, unsupported binaries — which is the fastest path from a number to a cause.

flowchart TD
  A[Count files on disk<br/>find -type f, wc -l] --> B[Ask the AI to<br/>count what it sees]
  B --> C{Numbers agree?}
  C -->|No| D[Diff the two<br/>file lists with comm]
  C -->|Yes| E[Spot-check the<br/>edge files by hand]
  D --> F[Name the blind-spot<br/>class it fell into]

Figure: The smoke-test loop — count the files on disk, ask the AI to count what it sees, and compare. If the numbers disagree, diff the two lists and name the blind spot that dropped each file; if they agree, spot-check the edge cases by hand, because agreement is necessary but not sufficient.

The five blind spots the gap reveals

A count mismatch tells you a gap exists; it does not tell you why. Five mechanisms cause almost every silent drop between the files on disk and the files an AI reads: symbolic links, dot-files and ignore rules, text encoding, binary attachments, and file permissions. Each one is a documented default, not a bug.

1. Symlinks. A default find walk does not follow symbolic links, and neither do many indexers. POSIX documents the opt-in: the -L option causes "the file information and file type evaluated for each symbolic link" to be "those of the file referenced by the link, and not the link itself." ⁵ So a folder of symlinks into another drive counts as a handful of links, not the hundreds of notes behind them. Compare the two walks:

find ~/notes -type f | wc -l
find -L ~/notes -type f | wc -l

A jump between those numbers is the size of your symlink blind spot.

2. Dot-files and ignore rules. Many tools skip hidden files and honor ignore lists by default. ripgrep's documentation states it plainly: "By default, ripgrep will respect gitignore rules and automatically skip hidden files/directories and binary files." ⁶ That default is sensible for code search and quietly wrong for notes — a .daily/ folder or anything named in a .gitignore vanishes. The extension histogram from command 4 will not catch a whole hidden directory; list it explicitly to check.

3. Encoding. Tools assume UTF-8. The WHATWG Encoding Standard is explicit that "the UTF-8 encoding is the most appropriate encoding for interchange of Unicode, the universal coded character set." ⁷ A note saved as UTF-16 or Latin-1 either decodes into garbage or is skipped as "binary." Joel Spolsky's rule is older than most note apps and still binding: "It does not make sense to have a string without knowing what encoding it uses." ⁸ List the files that are not valid UTF-8:

find ~/notes -type f -exec sh -c 'iconv -f UTF-8 -t UTF-8 "$1" >/dev/null 2>&1 || echo "$1"' _ {} \;

Every path it prints is a file your tool may be reading wrong, or not at all. When a file genuinely is in another encoding, a tool like ripgrep can be told to read it — it "supports searching files in text encodings other than UTF-8, such as UTF-16, latin-1, GBK, EUC-JP, Shift_JIS and more" ⁹ — but only if you know the file is there to configure it.

4. Binary attachments. A scanned PDF, a screenshot, an .xlsx — these show up in the count and carry real content, but a text-only RAG pipeline ingests the filename and nothing inside. This is a coverage gap that no amount of chunking fixes, because the bytes were never text. Command 4's histogram is how you find it: any non-text extension near the top of the list is content the model cannot see without a separate extraction step.

5. Permissions. A file the indexing process cannot read is a file the AI cannot read. POSIX defines the read bits by class — 0400 for owner, 0040 for group, 0004 for other ¹⁰ — so a note saved chmod 600 under one account is invisible to an indexer running as another. Find the owner-unreadable files portably:

find ~/notes -type f ! -perm -u+r

The GNU ! -readable test is more accurate because it accounts for the current user and ACLs, but it is a GNU extension, not POSIX — the ! -perm -u+r form above runs on macOS and BSD find too.

What the smoke test does not do

This is a coverage check, not a fix and not a security control. It finds the files an AI silently skips; it does not make the AI read them, and it does not stop an over-permissioned agent from reaching files it should not. Knowing the set is hygiene. Constraining the set is a separate discipline.

The distinction matters because the opposite failure is real too. OWASP's LLM06:2025 entry names "Excessive Agency," whose "root cause" is "typically one or more of: excessive functionality; excessive permissions; excessive autonomy." ¹¹ A coverage audit answers "is the AI seeing everything I meant to hand it?" — it says nothing about whether the AI can reach files you did not mean to hand it.

Treat this test as a smoke detector, not a lock. It tells you the room is filling with smoke; it does not put out the fire, and it does not decide which rooms the agent may enter.

Where to go once you find a gap

Discovery is the first half; the remedy depends on the blind spot. A wrong encoding needs a re-save; an ignored dot-folder needs a scope change; a note the model chunked badly needs restructuring. The fix always lives one level up — in how you shape the folder, the file, and the map you hand the agent.

Each of those fixes has its own method:

• If the gap is which files sit under the agent's view, shape the folder you hand your AI so the root matches your intent.
• If a note is present but the wrong parts surface, see how you split a note decides what an AI finds in it.
• If the agent wanders past the notes that matter, give it a map of your notes.
• And where this test asks what the AI can't see at all, its companion asks the opposite for a single session: which files did your AI actually read.

The ground-truth walk itself is cheap enough to run every time. ripgrep's benchmark table clocks a whitelist search at 0.063s against GNU grep's 0.674s, a 10.69× difference ¹² — so walking the files directly costs less than the doubt it removes.

Frequently asked questions

The questions below come from people running a local AI over a notes folder and getting answers they can't fully trust. Each answer is scoped to the smoke test — what the commands reveal, and what they deliberately leave to a separate fix.

How do I know what my local AI can actually see in my notes folder?

Count the files on disk with find ~/notes -type f | wc -l, then ask the AI to list and count the files it can see, and diff the two. The number you control is the ground truth; the AI's number is what it actually received. The gap between them is the set of files it cannot read.

What does my local RAG not retrieve?

Two different sets. First, files it never ingested — symlinked, hidden, wrongly encoded, binary, or permission-denied — which the five-command diff exposes. Second, files it ingested but ranked too low to surface, which is a retrieval-tuning problem, not a coverage one. This test finds the first set; the second needs evaluation, as Simon Willison argues below.

Why can't my AI find a note I know is there?

Because a filter dropped it before the model saw it. The note is on disk, but the tool's walk skipped it — most often a hidden folder, a non-UTF-8 encoding, a file type it does not parse, or a permission it lacks. Run command 3's comm diff to see the exact paths, then command 5 to get the tool's own reason.

Does find follow symlinks by default?

No. A default find walk evaluates the link itself, not its target. To follow symbolic links you pass -L, which POSIX defines as evaluating "the file referenced by the link, and not the link itself." ⁵ Diff find ~/notes -type f against find -L ~/notes -type f to measure how many files hide behind links.

Why does my RAG skip hidden files and dot-folders?

Because skipping them is the default in most search tooling. ripgrep, for example, "will respect gitignore rules and automatically skip hidden files/directories and binary files." ⁶ That default suits source code and misfits notes, where a .daily/ or .archive/ folder is real content. Check hidden directories by name; the extension histogram alone will not reveal a whole skipped folder.

Why won't my indexer read my UTF-16 or non-UTF-8 notes?

Because tools assume UTF-8, "the most appropriate encoding for interchange of Unicode." ⁷ A UTF-16 or Latin-1 file decodes wrong or is treated as binary and skipped. As Joel Spolsky put it, "it does not make sense to have a string without knowing what encoding it uses." ⁸ List the offenders with the iconv one-liner in blind spot 3, then re-save them as UTF-8 or configure the tool's encoding.

How do I count the files in a directory on the command line?

Use find ~/notes -type f | wc -l. The -type f primary matches regular files, so directories are excluded ⁴, and wc -l counts the lines find prints — one per file. This is the ground-truth number the whole smoke test diffs against, and it is worth writing down before you ask the AI anything.

Good retrieval is not something you assume; it is something you check. As Simon Willison writes, "good RAG systems are backed by evals — it's extremely hard to iterate on and improve a system like this if you don't have a good mechanism in place to evaluate if your changes are making things better or not." ¹³

A file count is the smallest eval there is, and it catches the largest class of silent failure. An AI's answer about your notes is only ever as complete as the set of files it managed to read — and the only way to know that set is to count it yourself.

Notes kept as plain files on your own device make that count trivial: the folder you walk with find is the same one the AI reads — the quiet advantage of writing in something like mnmnote.com.