Build a Personal RAG You Can Actually Audit

A personal RAG is auditable when two things hold: every answer cites the exact note it drew from, and the system refuses to answer when no note supports it. The first lets you check the source — the second stops the model from filling silence with invention. Plain files you own make both honest.

The pattern is older than the hype. When Google launched NotebookLM in 2023, Raiza Martin and Steven Johnson wrote that the tool lets you "ground" the language model in your notes and sources, and that "we make that fact-checking easy by accompanying each response with citations, showing you the most relevant original quotes from your sources" ¹. That is the contract. The rest of this post is how to hold a personal version of it to account — and where it quietly breaks.

What makes a personal RAG auditable?

A personal RAG is auditable when you can trace any answer back to the specific note that produced it and verify the model did not invent the rest. Auditability is not accuracy. It is the property that lets you check accuracy yourself — you see which file answered, open it, and confirm the claim is really there.

The demand is real and rising. The open-source project lfnovo/open-notebook, described as "An Open Source implementation of Notebook LM with more flexibility and features," has passed 30,000 stars on GitHub ². reorproject/reor, a "Private & local AI personal knowledge management app," has passed 8,000 ³. People want AI over their own notes. The unclaimed part is trust — not can it answer, but can I see which note answered.

Why "just ask it to cite its sources" is not enough

Telling a language model to cite its sources is necessary and nowhere near sufficient. Left to its own narration, a model produces citations that look right and are not. The trustworthy version enforces the citation as part of generation — rather than requesting it as a polite afterthought.

The failure is documented. Qi, Sarti, Fernández, and Bisazza, introducing the MIRAGE attribution method, observe that "self-citing LLMs often struggle to match the required format, refer to non-existent sources, and fail to faithfully reflect LLMs' context usage throughout the generation" ⁴. A citation a model writes for itself can point at a source that was never retrieved. Their method instead "detects context-sensitive answer tokens and pairs them with retrieved documents contributing to their prediction" ⁵ — attribution derived from what the model actually used, not what it claims it used.

So the design question is not "did it cite?" It is "is the citation enforced, and does it correspond to a document the system genuinely retrieved?"

The three-step contract: retrieve, cite, verify

A personal RAG you can audit runs three steps in order, and each step is checkable. Retrieve the candidate notes for a question. Generate an answer that cites the exact notes used. Then verify the cited notes really say what the answer claims — by opening them. Skip any step and the audit trail breaks.

The flow below is the whole method on one screen.

flowchart TD
  A[Your question] --> B[Retrieve candidate notes]
  B --> C{Anything<br/>relevant?}
  C -->|No| D[Refuse / say<br/>nothing found]
  C -->|Yes| E[Answer + cite<br/>the exact note]
  E --> F[Open the cited<br/>note and check]

Figure: The retrieve-cite-verify contract. A question retrieves candidate notes; if nothing relevant is found the system refuses; if something is found, it answers with a citation to the exact note, which you then open and check yourself. The refuse branch is what stops the model inventing an answer when your notes are silent.

Step 1 — Retrieve from your own notes only

Retrieval is the gate. The model should see only your notes for the question at hand, scoped to a known set of files, so that every later citation has a real document behind it. If retrieval pulls from the model's training memory instead of your corpus, no citation can be honest, because there is no file to open.

This is the part most "chat with my notes" setups get right by default — embeddings, keyword search, or a hybrid over your files. The mechanism matters less than the boundary. Answers come from the retrieved set, full stop.

Step 2 — Cite the exact note, enforced

The answer must name the specific note it used, and the citation must be enforced during generation rather than narrated after the fact. This is the half that turns a confident paragraph into a checkable claim — the difference between a sentence you trust and a sentence you can check. A citation-enforced system, in the words of one fiscal-document study, "enforces citations during generation, and supports abstention when evidence is" insufficient ⁶.

Enforcement is what separates this from step-two theater. Asking nicely produces the non-existent sources MIRAGE warns about ⁴. Enforcing the citation against the retrieved set produces a pointer you can follow.

Step 3 — Refuse when nothing supports the answer

The system must be willing to say it found nothing. A model that always answers will always invent when your notes are silent, and an invented answer is the one failure a citation cannot catch — there is no note to open. Refusal is the other half of trust.

Song and colleagues, measuring RAG trustworthiness, found that their Trust-Align method "also significantly enhances models' ability to correctly refuse and provide quality citations" ⁷, treating refusal and citation as the same skill. They frame the broader problem plainly: "there is a gap in understanding the appropriateness of LLMs for the RAG task" ⁸. A personal RAG that never refuses has not closed that gap — it has hidden it.

A citation proves retrieval, not correctness

A citation proves the answer came from a note, not that the answer is right. The model can cite the correct note and still misread it — invert a number, miss a negation, summarize a nuance away. The citation tells you where to look. It does not look for you.

Google's own NotebookLM team said this in the launch post: "While NotebookLM's source-grounding does seem to reduce the risk of model 'hallucinations,' it's always important to fact-check the AI's responses against your original source material" ⁹. The vendor that built the cite-back feature tells you to check the source anyway. That instruction is the point of step three, made into a habit. A cited answer is a starting place for your judgment, not a replacement for it.

This is also why auditability and privacy are different properties. A hosted model still reads the note contents it retrieves over; the audit win is that you can verify the answer, regardless of where the model runs. If you also care about keeping the note contents on your machine, that is a separate decision — about local models and where data goes.

Why plain files make the audit real

The cite-back contract is only as honest as the thing the citation points at. When your notes are plain Markdown files you own, a citation back to a note points at a file you can open and read yourself — not a vendor's opaque index you take on faith. The audit becomes a click, not a support ticket.

Steph Ango, who leads Obsidian, put the durability case in one line: "Apps are ephemeral, but your files have a chance to last" ¹⁰. The same logic governs auditability. A citation to a file that outlives its app is a citation you can still check next year. A citation into a proprietary store is only as auditable as that store's willingness to show you the underlying document — which is to say, not yours to verify.

You can see the pattern already working in the wild. NotebookLM accompanies answers with quotes from your sources ¹. Open-source notebooks like open-notebook reproduce the cite-back behavior over files you control ². Local-first tools like reor keep both the notes and the model on your machine ³. None of these is the only way; together they show that "answer cites the exact note, over files you own" is a design you can hold to account, not a wish.

Common mistakes when building a personal RAG you trust

Most failures of trust come from skipping one of the three steps or confusing two properties. A system that cites is not automatically a system you can check — and a system you can check is not automatically a private one. The mistakes below are the ones that turn an audit trail into a comfort blanket.

• Treating "it cited something" as proof it is right. Citation is retrieval, not correctness ⁹. Open the note and read the relevant line before you rely on the answer.
• Asking for citations instead of enforcing them. A self-narrated citation can name a source that was never retrieved ⁴. Prefer a setup that enforces the citation against the retrieved set ⁶.
• Letting the model answer when nothing matched. A RAG that never refuses invents on empty retrieval. Refusal is a feature, not a bug ⁷.
• Confusing auditability with privacy. A cited answer can still be processed by a hosted model that sees your note contents ⁹. Decide where the model runs separately.
• Citing into something you cannot open. A pointer into an opaque index is not an audit. Keep the source a file you can read ¹⁰.

How this looks in MNMNOTE

When your notes are plain Markdown stored locally in your browser, the cite-back contract has its most honest form: the citation names a file you can open and read on the spot. The answer points at your note — not a server's summary of it. That is an architectural property of owning open files, not a feature toggle.

If you want the parent argument for why plain files are good AI source material, the companion post on why Markdown notes are AI-ready covers the why; this post is the trust layer on top — how to verify which file answered.

Frequently Asked Questions

How do I know which of my notes the AI's answer came from?

You know when the answer carries a citation to the exact note and the system enforces that citation during generation rather than narrating it afterward ⁶. Open the cited note and confirm the claim is there. If your notes are plain files you own, the citation points at something you can read yourself.

Can ChatGPT or NotebookLM hallucinate even when it cites a source?

Yes. A citation proves the answer was retrieved from a note, not that the answer is correct — the model can cite the right note and still misread it. Google's own NotebookLM team advises that "it's always important to fact-check the AI's responses against your original source material" ⁹. Always open the source.

Is asking the model to "cite its sources" enough?

No. Self-citing models "refer to non-existent sources" and fail to reflect what they actually used ⁴. A polite request is not enforcement. Trust comes from citations enforced against the retrieved set — plus the ability to refuse when nothing matched ⁶⁷.

What does it mean for a personal RAG to "refuse"?

Refusal means the system answers "nothing in your notes supports this" instead of inventing. It is the failure a citation cannot catch, because an invented answer has no note to open. Research treats refusal and citation as the same trust skill ⁷.

Does an auditable RAG also mean a private one?

No — they are separate properties. Auditability is whether you can verify the answer's source; privacy is whether the note contents stay on your machine. A hosted model still reads what it retrieves ⁹. Decide where the model runs as its own question.

Why do plain files matter for auditability?

Because the citation is only as honest as what it points at. A citation into a file you own and can open is checkable; a citation into a vendor's opaque index is only as auditable as the vendor allows. "Apps are ephemeral, but your files have a chance to last" ¹⁰, and so do the citations into them.

A citation is a promise you can keep only if you can open what it points at. Build your personal RAG so every answer names the note and refuses when none fits, keep those notes as plain files you own, and the promise holds — your notes live in your browser at mnmnote.com, where every answer can point back at a file you can read.