Make Your AI Say 'It's Not in Your Notes' — the Abstention Discipline for a Personal RAG

Make your AI admit when your notes don't contain the answer instead of inventing one. The discipline has three parts: a strict instruction to answer only from your notes, a fixed refusal phrase the model must emit when they fall short, and a grep double-check that confirms the missing term really is absent. It reduces confabulation — it does not eliminate it.

Knowing when to stay silent is a measured, unsolved capability, not a personality trait you can prompt into existence. Meta's FAIR team put it plainly in AbstentionBench, presented at NeurIPS 2025: "For Large Language Models (LLMs) to be reliably deployed in both everyday and high-stakes domains, knowing when not to answer is equally critical as answering correctly."¹ The same paper is honest about the ceiling — "while a carefully crafted system prompt can boost abstention in practice, it does not resolve models' fundamental inability to reason about uncertainty."² So the instruction below is necessary and it works, but it is not a guarantee. The whole method is built around that limit: instruct the model to abstain, then verify the abstention against your actual files rather than trusting it.

The one habit: answer only from the notes, or say one fixed phrase

The single habit is to forbid the model from answering out of its own memory. Give it one rule: use only the retrieved notes, and when they fall short, reply with one exact phrase. That fixed phrase is the whole trick — a token like NOT IN YOUR NOTES is a signal you can detect, count, and grep.

This matters because retrieval failure is silent. When your vault has no relevant note, the search step does not raise an error — it hands the model a thin or off-topic context, and a capable model fills the gap from training. Google and UC San Diego researchers measured exactly this in "Sufficient Context": larger models "excel at answering queries when the context is sufficient, but often output incorrect answers instead of abstaining when the context is not."³ The instruction sets the behavior. The fixed phrase makes the behavior auditable.

Why the default is to invent an answer

A default RAG has no concept of an empty answer. Retrieval returns its top matches whether or not any of them are relevant, and the model treats whatever arrives as permission to respond. Making it refuse is not the natural state — it is a behavior you engineer, because scaling up does not add it and reasoning training can subtract it.

AbstentionBench is blunt on that point: "Evaluating 20 frontier LLMs reveals abstention is an unsolved problem, and one where scaling models is of little use."⁴ Worse, the fix people reach for first can backfire — the same study found that "reasoning fine-tuning degrades abstention (by 24% on average), even for math and science domains on which reasoning models are explicitly trained."⁵ A "smarter" model is often more willing to answer, not less.

The empirical picture at the note-sized scale is the same. On Lech Mazur's confabulation leaderboard, 201 questions "confirmed by a human to lack answers in the provided texts" were run at "a temperature setting of 0."⁶ Even under those controlled conditions, confabulation ran from roughly 10.9% for the best-scoring model to 66.3% for the weakest in the set.⁷ Read those numbers with the benchmark author's own caveat attached: "the absolute percentage should not be used to infer that LLMs frequently confabulate. This leaderboard does not reflect a 'typical' hallucination rate."⁸ The set is adversarial by design. The lesson is not a rate — it is that the unprompted default is not safe, so the safe behavior has to be asked for and then checked.

The five-minute version: instruction, refusal token, grep check

Add three things to a RAG you have: a system instruction that pins the model to the retrieved notes, a fixed refusal phrase it must return when those notes fall short, and a one-line search, run on every refusal, that confirms the term is truly absent from your files. The instruction is the paste-in below; the checks are yours.

flowchart TD
  A[Your question] --> B[Retrieve from<br/>your notes]
  B --> C{Notes contain<br/>the answer?}
  C -->|No| D[Reply exactly:<br/>NOT IN YOUR NOTES]
  C -->|Yes| E[Answer, and quote<br/>the note]
  D --> F[Grep the term<br/>across your files]
  F --> G{Term really<br/>absent?}
  G -->|Yes| H[Refusal was correct]
  G -->|No| I[Retrieval missed it:<br/>fix the retriever]

Figure: The abstain-or-answer gate, plus the grep double-check. A question is answered only when the retrieved notes contain the answer; otherwise the model returns the fixed refusal token. Every refusal is then verified by searching your files: if the term is genuinely absent the refusal was correct, and if it turns up, retrieval missed an existing note — a retriever bug, not a model one. The check is what turns a refusal from a claim into something you can trust.

Start with the instruction. It is our template, grounded in the abstention research above, and it carries the honesty caveat in its own wording:

You answer only from the NOTES provided below. Treat the notes as your
only source of truth. Do not use outside knowledge.

If the notes do not contain enough information to answer, reply with
exactly this line and nothing else:

NOT IN YOUR NOTES

When you do answer, quote the sentence from the notes that supports it.

NOTES:
{retrieved_notes}

QUESTION:
{question}

Then wire the check around it:

1. Pin the model to the context. Pass the retrieved notes as the only source and paste in the instruction above. No outside knowledge, one job.
2. Pick one unmistakable refusal token. NOT IN YOUR NOTES in all caps is easy to match and unlikely to appear by accident. Use the same string everywhere.
3. Ask something you know is absent. Query a topic your vault has never covered. If the model answers anyway, tighten the instruction before you trust anything else.
4. Grep every refusal. When the reply is the token, run rg -i "term" ~/notes. An empty result confirms the refusal was correct. A hit means the term was there and retrieval missed it.
5. Route the two failures differently. A correct refusal needs no fix. A grep hit is a retrieval bug — your search missed a note that exists — and it is fixed in the retriever, not the prompt.
6. Log refusals with their queries. A short log of what the model declined to answer is the raw material for the regression suite below.

The thirty-minute version: a known-absent regression suite

The upgrade is a small set of questions you know your vault cannot answer, each paired with the assertion that the model returns the refusal token. Run it after every prompt change or model swap. It converts "does it abstain?" from a hope into a check that reruns in seconds and fails loudly when a change re-enables confabulation.

Keep the suite in a plain file beside your notes. Each entry is a question plus the expected outcome — abstain — and a grep term to confirm the absence is real, not a retrieval miss:

# known-absent.md — questions the vault must refuse
# expect: model returns exactly "NOT IN YOUR NOTES"

- q: "What did I decide about the Helsinki contract?"
  grep: "helsinki"        # confirm no such note exists
- q: "Summarize my notes on quantum error correction."
  grep: "quantum error"
- q: "What's my account number at First Meridian Bank?"
  grep: "first meridian"

The assertion loop is small: for each entry, send the question through the pinned instruction, check the answer equals the refusal token, then run the grep to confirm the term truly is absent. If a probe ever starts getting a real answer, either the model stopped abstaining or you actually added a note on that topic — and the paired grep tells you which. This is the negative-path twin of burying a canary phrase to prove retrieval fires on content that is present; here you prove the model stays quiet on content that is not.⁹

Common mistakes

Most abstention failures are self-inflicted, and they cluster around one error: treating the instruction as a guarantee and skipping the check. The instruction reduces confabulation; the verification is what makes a refusal trustworthy. Each mistake below ends the same way — a confident answer no note supports, or a refusal you had no way to trust.

• Trusting the instruction alone. A crafted system prompt "does not resolve models' fundamental inability to reason about uncertainty."² It is necessary, not sufficient — which is the entire reason for the grep double-check and the regression suite.
• Using a free-form refusal. "I don't have enough information" is unmatchable and drifts phrasing every run. Without one fixed token you cannot detect, count, or grep the refusals, and the whole audit collapses.
• Never checking the refusal against the files. A refusal can be wrong: the note exists but retrieval missed it. Only rg over your actual notes tells you whether "not in your notes" was true or a retrieval bug wearing its costume.
• Confusing an empty vault with the wrong note. If a note was retrieved but it's the wrong one, that's a grounding problem, not abstention — the fix is making the model quote the note before you believe it, which a companion piece covers in full.¹⁰
• Selling abstention as elimination. Refusal "prevents hallucinated content but does not enhance factual knowledge,"¹¹ and the model can still ignore the rule and answer anyway. Abstention lowers the confabulation rate; it never drives it to zero.

How this works with your own notes

This discipline is only auditable when your notes are files you can search. When every note is plain Markdown stored locally on your device, an "it's not in your notes" refusal is checkable in one command — rg -i "term" ~/notes — and the empty output is the proof. A black-box assistant gives you the same refusal with nothing to verify it.

That verifiability is the payoff of open, greppable files. Rejecting the unanswerable is a named capability that RAG systems lack by default — a 2025 preprint on realistic multi-hop queries argues "RAG systems must be able to reject unanswerable, out-of-scope queries and identify failures of retrieval and multi-hop reasoning."¹² When your corpus is small, plain full-text search sidesteps the whole problem, because grep never confabulates an answer it cannot find.¹³ And when near-duplicate, almost-right notes are the issue rather than an empty vault, that is a distinct failure mode with its own fix.¹⁴ One scope note on privacy: the retrieval and the grep read local files, but if you send note text to a third-party model to generate the answer, that text leaves your device for your chosen provider — the auditable part is the abstention and the check, not a claim that nothing ever leaves. Keeping the answer honest is the same instinct as keeping the whole retrieval layer one you can inspect.¹⁵

Frequently asked questions

How do I stop my AI from making things up about my notes? Pin the model to the retrieved notes with a strict instruction, and require a fixed refusal phrase when they don't contain the answer. Then verify: on every refusal, grep your files for the term. The instruction reduces invented answers; the grep check proves each refusal was earned rather than trusting the model.

How do I make my RAG say "I don't know" instead of hallucinating? Give it one exact phrase to say — NOT IN YOUR NOTES — and forbid outside knowledge in the system instruction. A fixed token beats a vague "I'm not sure" because you can detect and count it. This boosts abstention in practice, though it cannot fully resolve a model's difficulty reasoning about its own uncertainty.²

Why does the model answer when the information isn't in my documents? Because retrieval fails silently and the model treats whatever context it gets as permission to respond. Researchers found that capable models "output incorrect answers instead of abstaining when the context is not" sufficient.³ The empty-set case is not the model's natural instinct to catch — you have to instruct it, then check the result.

Can I check whether the AI was right to refuse? Yes, and that is the point of using plain files. When a note lives as Markdown on your own device, rg -i "term" ~/notes confirms in one line whether the term is genuinely absent. Empty output means the refusal was correct; a hit means retrieval missed an existing note and needs fixing.

Does telling the model to only answer from context fix hallucination? It reduces it, not eliminates it. A crafted system prompt "can boost abstention in practice" but "does not resolve models' fundamental inability to reason about uncertainty,"² and refusal "does not enhance factual knowledge."¹¹ Treat the instruction as a strong default and the grep check plus regression suite as the parts you actually trust.

What's the difference between abstention and grounding? Abstention is upstream: there is no relevant note, so the correct output is a refusal. Grounding is downstream: a note was retrieved and the model must quote a real span from it. This piece is about the empty set; making the model quote its source is the separate grounding check.¹⁰

Is this the same as testing whether retrieval fired? No — it's the opposite path. Testing retrieval checks that a phrase which is in your notes gets found; a canary phrase proves the positive case.⁹ Abstention checks that a phrase which is not there produces a refusal. Both belong in a healthy RAG, and the two suites read as mirror images of each other.

Won't this make the model refuse everything? No. Guided abstention trades a few confident wrong answers for a more trustworthy set, rather than muting the model. In the Sufficient Context study, it "improves the fraction of correct answers among times where the model responds" by 2 to 10% for the models tested.¹⁶ You lose some answers and gain reliability on the ones that remain.

A model that will not say "I don't know" is not careful — it is only confident. The fix is not a smarter model but a rule it must obey and a file you can search to confirm it obeyed. Teach it the phrase, then verify the phrase was earned.

MNMNOTE keeps your notes as plain Markdown on your own device, so an "it's not in your notes" answer is one you can check for yourself — mnmnote.com.