Make the AI Quote Your Note Before You Believe It

When an AI answers from your notes, do not trust the claim — make it return the exact sentence it used, plus the note that sentence lives in. Then search your files for that sentence. If the quote is nowhere on disk, the answer was not grounded in your notes. Discard it.

That is the whole loop, and it takes about ten seconds. It works because you own the files. A quoted span is a string, and a string either exists in your folder or it does not — a check your file system answers in one command, with no cloud, no model, and no trust required.

Simon Willison made the same argument about code. "The moment you run LLM generated code, any hallucinated methods will be instantly obvious: you'll get an error," he wrote in March 2025 ¹. "With code you get a powerful form of fact checking for free."

Prose over your notes has no compiler. This loop gives it one: the grep either finds the quote or it does not, and the finding replaces the "critical eye" and "well developed fact checking skills" Willison says prose otherwise demands ¹.

The single habit: quote first, then grep

The habit is one rule applied to every answer: no claim without a quotable source. Make the AI copy the exact words from your note and name the file, then run a literal string search for those words. Grounding is not something the AI asserts — it is something your file system confirms.

You are checking a specific, narrow property. Researchers call it faithfulness. The RAGAS evaluation framework defines it as the idea that "the answer should be grounded in the given context" ² — the claim must be inferable from the source it cites, not merely plausible. The grep is the crudest possible faithfulness check, and its crudeness is the point: a literal match cannot be argued with.

The base rate is why this matters. When researchers built ALCE, a benchmark for AI systems designed to cite their sources, they found that "on average, even the best models lack complete citation support 50% of the time" on one dataset ³. A system engineered to attribute its claims still fails to fully support half of them. A general chat model reading your notes was not engineered for this at all.

One assumption sits underneath the whole loop: that the AI is reading your notes at all. If you are not sure retrieval even fires, bury a canary in your notes is the upstream test — plant a fact only your notes contain and see if it comes back. This loop is the next step: not did retrieval fire, but is the answer it produced actually in your notes.

The ten-second version

Five moves, no dependencies, about ten seconds after the answer lands. The one non-obvious move is the first: you have to demand the quote before you read the answer, because an answer you have already believed is one you will rationalize.

1. Set the rule up front. Tell the AI: for every factual claim, return the exact sentence from my notes that supports it, in quotation marks, with the file name. No quote, no claim.
2. Ask your question. Let it answer in that format — claim, then the verbatim span it rests on, then the file.
3. Copy one quoted span. Take the exact words it attributed to your note, punctuation and all.
4. Search your files for it, literally:

rg -F "the exact sentence the AI quoted" ~/notes

The -F flag matters. ripgrep's guide describes it plainly: "Disable regular expression matching and treat the pattern as a literal string" ⁴. Your notes are full of parentheses, periods, and asterisks; without -F, a quoted span is read as a regular expression and the search breaks on the first special character. The identical flag exists in plain grep -F, so no new tool is required.

5. Read the result. One file, exact match: the quote is real, and the claim is at least grounded in your note. Zero matches: the span is not in your files, so the AI invented it or paraphrased it into something you never wrote. Treat an unlocatable quote as an ungrounded claim and drop it.

Why a string match beats a careful read

Because a careful read is exactly the skill that fails at scale, and a string match does not get tired. The measured gap between a citation and the claim it supposedly supports is large and well documented — the check has to be mechanical to survive contact with it.

Stanford researchers evaluating generative search engines put a number on the gap. Across the systems they tested, "a mere 51.5% of generated sentences are fully supported by citations and only 74.5% of citations support their associated sentence" ⁵.

Read the second figure slowly: roughly one cited sentence in four does not actually say what it was cited for. The citation looks like support; it is not. This is why you grep the quoted sentence, not the file name — a correct-looking source pointer is not evidence the words support the claim.

The same authors define the standard the check enforces. A trustworthy system, they write, should "cite comprehensively (high citation recall; all statements are fully supported by citations) and accurately (high citation precision; every cite supports its associated statement)" ⁵. Precision is the half a lone file name cannot prove. Only the literal span can.

The manual grep is the floor of this idea, not the ceiling. RAGAS turns the same faithfulness question into an automatable score, judging whether a claim can be inferred from its retrieved context rather than only whether a string appears ².

That is worth knowing for two reasons. It confirms the check is sound enough to formalize — you are not improvising. And it marks where the crude version stops: a literal match catches the blatant miss, the quote that was never there, but it will not weigh a paraphrase that is technically supported. For that judgment there is no shortcut. You read the note.

flowchart TD
  A[AI answers from<br/>your notes] --> B[Demand a verbatim<br/>quote + source file]
  B --> C{Grep finds the<br/>quote on disk?}
  C -->|No| D[Ungrounded —<br/>discard the claim]
  C -->|Yes| E[Grounded: quote<br/>is in your note]
  E --> F{Is the note<br/>itself correct?}
  F -->|Separate check| G[Verify the note's<br/>own accuracy]

Figure: the grounding loop. The AI answers, then must return a verbatim quote and the file it came from. You search your files for that quote. No match means the claim is ungrounded — discard it. A match proves only that the quote is in your note; whether the note itself is correct is a separate check the grep never performs.

What this proves — and what it doesn't

It proves the quote is in your note. It does not prove the note is right. Those are two different questions, and collapsing them is the one mistake that turns a useful check into false confidence. A found quote means the AI read your file faithfully. It says nothing about whether your file was true.

Maynez and colleagues drew this line precisely in their 2020 study of summarization. They define a factual hallucination as content that "contains information not found in D that is factually correct" ⁶ — true, but not in the source. So a claim can be true yet unfaithful, and, running the other way, faithful yet false: perfectly grounded in a note that was wrong when you wrote it. The grep catches the second case and is blind to the first.

This is not a small caveat. It is the boundary of the whole method. The loop answers "did the AI take this from my notes?" It cannot answer "are my notes correct?" That second question is about the truth and currency of the file itself — a different job, handled by dating your notes so the AI knows which one is current. Grounding and correctness are separate audits. This one is grounding.

A fair objection: these studies measure research summaries and generative search engines, not your personal vault. True. They are not a measurement of your files — they are a base rate for how often generated text drifts from the source it claims. In one summarization study, hallucinations showed up in more than 70% of single-sentence summaries ⁶; that base rate is high enough to justify checking every time. The number you should trust about your own notes is the one your own grep returns.

Common mistakes

Four errors account for nearly every false result, and three of them happen before you run a single search. Each one produces a confident-looking outcome that means nothing — a green light you did not earn, or a red light that only proves your search was built wrong. Rule them out in order.

• Accepting a paraphrase. If the AI hands you a smooth summary instead of a quotation, there is nothing to match. An abstractive paraphrase cannot be string-searched. Reject it and demand the extractive form — the exact words — before you grep.
• Grepping the file name, not the sentence. A correct source pointer is not evidence. Stanford's 74.5% figure is the whole reason ⁵: the cited file can be right while the quoted words do not support the claim. Search the sentence.
• Dropping the -F flag. Without literal-string mode, a quote containing (, ., or * is parsed as a regular expression and silently mis-matches. A "no result" then means your search broke, not that the quote is missing.
• Treating a match as proof the answer is correct. A found quote is grounding, not truth. Maynez's faithful-yet-false case is real ⁶: the note can be wrong. Verify the note separately.

How this works in a plain folder of notes

The loop needs one thing from your notes: they have to be real files you can point a command at. If your notes are plain Markdown in a folder you own, the source of truth sits on your own device, and confirming a quote is one line of search. No export, no API, no plugin required.

This is the quiet reason open formats matter here. You cannot run a literal string search against an opaque cloud store or a vector index you are not allowed to read — you would be asking the same system that produced the answer to also certify it. Plain files break that circle. The verification runs on something other than the model under test.

One honest scope note. The grep runs entirely on your own files, offline. The AI answer did not: if you brought your own key and sent note text to a third-party model to produce that answer, the text you chose to send left your device on that path. The verification is local; the generation used whatever provider you picked.

Keep the two straight, and the check stays trustworthy — because it never depends on the model's account of itself. Proving which note answered is a companion habit: saving the AI answer with its sources and running AI over a local, private folder of notes both build on the same open-file foundation.

Frequently asked questions

These are the questions people actually type when they suspect an AI is answering from thin air instead of from their own files. Each answer below is short enough to act on, and each points back to the section above where the underlying evidence lives.

How do I check that an AI answer is actually supported by my own notes and not hallucinated? Force a quote. Tell the AI to return, for every claim, the exact sentence from your notes plus the file name, then run rg -F "that sentence" ~/notes. One exact match means the claim is grounded in your note. Zero matches means it was invented or paraphrased — discard it.

Is the AI making this up? Sometimes, and this tells you which times. If the quoted sentence it attributes to your notes does not appear anywhere in your files, that specific claim is unsupported. A benchmark built to cite sources still failed to fully support half its claims ³, so assume nothing until the span locates.

How do I check if the AI actually read my notes? Demand a verbatim span, not a summary, and search for it. A model that read your file can quote it exactly; one that is guessing produces a plausible sentence that greps to zero results. The literal match is the difference between reading and inventing.

The AI cited a source that doesn't say that — why? Because citations miss more often than people expect. Stanford found only 74.5% of citations actually support their associated sentence ⁵ — roughly one in four does not. That is why you grep the exact quoted sentence, not just the filename it pointed at.

Does grounding mean the answer is correct? No. Grounding means the quote is genuinely in your note. Correctness means the note itself is right — a separate question. Maynez and colleagues show a claim can be faithful to a source yet still false ⁶. The grep proves the first; it cannot prove the second.

What if the AI paraphrases instead of quoting? Reject the paraphrase. A rewritten summary cannot be string-matched, which defeats the check. Instruct the AI to return the extractive form — the exact words from the note — first. If it cannot produce a span that greps, treat the claim as unsupported.

Do I need ripgrep specifically? No. rg -F is fast on a large vault, but plain grep -F uses the identical literal-string flag and ships on every Unix-like system. The method depends on the literal match, not on any particular tool.

A grounded answer is not a correct answer — it is a checkable one, and checkable is the most any tool can honestly offer you about your own notes.

If you want notes a grep can actually confirm, mnmnote.com keeps them as plain Markdown files on your own device, where one line of search settles what the AI really read.