Your AI Is Just as Confident When It's Wrong About Your Notes

No. A language model states its confidence in the same fluent, self-assured tone whether it read your note correctly or invented the answer. That tone is a style it learned, not a measurement of accuracy. Treat every confident claim about your notes as a claim to verify — grep the source, don't trust the voice.

The gap between how sure a model sounds and how often it is right has a name in the research: calibration. In 2023, seven researchers led by Miao Xiong ran the first broad benchmark of it for large language models, later accepted at ICLR 2024. They tested confidence calibration and failure prediction "across five types of datasets (e.g., commonsense and arithmetic reasoning) and five widely-used LLMs including GPT-4 and LLaMA 2 Chat." ¹ Their headline finding is the one to sit with: "LLMs, when verbalizing their confidence, tend to be overconfident, potentially imitating human patterns of expressing confidence." ¹ Note the careful word: verbalizing. The study measures what a model says about its certainty, on public question-answering benchmarks — not a metaphysical inner sureness, and not your private vault. The mechanism, though, travels.

What the confident tone actually is

Most people read fluency as evidence. When a model answers in complete, assured sentences, the assurance feels like proof it checked the note and got it right. The instinct is reasonable: in people, confidence and competence usually travel together. In a language model, the two come apart, and the tone survives the mistake.

A model learned its voice from finished, edited prose — text written by people who had already done the work and knew the answer. So it inherited the register of someone who knows, and it uses that register whether or not it actually retrieved the right span from your file. Xiong's team read the pattern the same way: the overconfidence looks like the model "imitating human patterns of expressing confidence." ¹ The certainty is a style, copied from writing that earned it. Your note did not earn it.

Why stated confidence barely tracks correctness

The decoupling is measurable, not a hunch. When you ask a model how sure it is, the number runs systematically high and correlates weakly with whether the answer is true. Xiong's benchmark found verbalized confidence "overconfident" across every model tested, GPT-4 included. ¹ The signals underneath the words are barely better.

Here is the uncomfortable number. Trying to tell a model's right answers from its wrong ones, even by reading its internal probabilities rather than its spoken confidence, the researchers report an AUROC "gap is narrow, e.g., 0.522 to 0.605." ¹ An AUROC of 0.5 is a coin flip. So the best internal signal lands near 0.6, the spoken confidence near 0.52, and the distance between peeking inside the model and simply listening to it is small. Neither is far from chance at separating correct from incorrect.

That miscalibration is not a quirk of one model line. It is a documented property of modern neural networks in general. Back in 2017, Chuan Guo and colleagues at Cornell found that "modern neural networks, unlike those from a decade ago, are poorly calibrated." ² Their study was on image and document classifiers, not language models, so it does not speak directly to your notes. What it establishes is older and broader: as networks grew more capable, their confidence stopped matching their accuracy. Calibration is a known failure, studied for years — not a new surprise you are imagining.

What actually helps — and why it isn't enough

Confidence is not hopeless, and honesty requires saying so. Two findings cut the other way: a model carries some genuine self-knowledge, and simply asking it for a calibrated confidence improves the number. Both are real. Neither is strong enough to let you skip the check.

Start with the self-knowledge. An Anthropic technical report by Saurav Kadavath and colleagues, a 2022 preprint that was not peer-reviewed, found that "larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format." ³ Models do hold a partial signal about what they know. But the same paper is candid about the edge of that signal: the models "struggle with calibration of P(IK) on new tasks," where P(IK) is the model's own estimate that it knows the answer. ³ Partial, and format-dependent, and weakest on the unfamiliar — which is exactly where your idiosyncratic notes live.

Asking helps too. Katherine Tian and colleagues at Stanford, in work presented at EMNLP 2023, found that "verbalized confidences emitted as output tokens are typically better-calibrated than the model's conditional probabilities on the TriviaQA, SciQ, and TruthfulQA benchmarks, often reducing the expected calibration error by a relative 50%." ⁴ A 50% cut is large. It is also relative, measured down from a poorly-calibrated baseline, so better does not mean solved. Xiong's benchmark agrees on the ceiling: every mitigation it tried helped somewhat, yet "all investigated methods struggle in challenging tasks, such as those requiring professional knowledge." ¹

There is even a wrinkle worth keeping honest. Xiong found that for LLMs specifically, "as model capability scales up, both calibration and failure prediction performance improve." ¹ Bigger, stronger models are getting better calibrated, not worse — the opposite of Guo's older classifier result, which is why the two papers belong in separate lanes. The trend is encouraging. It is not a reason to trust a confident sentence about your own file today.

None of these papers tested a personal note vault. They measured trivia, reasoning, and true-or-false questions. Mapping the result onto "so a confident answer about your notes is equally unearned" is a reasoning step, not a finding the researchers reported. It holds because the mechanism is general: the model emits confident tokens from the register it learned, whether or not the retrieved context supports the claim. The benchmark just lets you watch it happen with a scorekeeper attached.

How to check a confident answer about your notes

You cannot make the tone honest, but you can make the claim checkable. The move is mechanical: force the model to quote the exact span it is relying on, then confirm that span exists in your file. Because your notes are plain Markdown on your own device, that confirmation is one command, run locally, with no round-trip to a vendor.

The habit is small enough to keep. It replaces a feeling ("that sounded right") with a test that returns yes or no.

1. Demand the quote. Ask for the verbatim sentence from your note, not a paraphrase or a summary. A model that cannot produce the span it claims to be citing is confabulating with confidence — this is the grounding check in make the AI quote your note before you believe it. ⁵
2. Grep the span. Run rg -F "the exact quoted sentence" vault/ against your notes. If it returns nothing, the confidence was unearned, no matter how sure the answer sounded. The file is the ground truth; the tone is not.
3. Separate confidence from agreement. A model also sounds certain while simply agreeing with a premise you handed it — a different failure worth its own guard, covered in ask a leading question about your notes and the AI will agree. ⁶
4. Prefer "not in your notes" to a fluent guess. An honest miss beats a confident invention; make abstention the expected answer when the vault is silent, as in make your AI say it's not in your notes. ⁷
5. Save the verified answer with its source. Once a claim survives the grep, store it with the note it came from. Don't re-ask and re-roll the dice on a fresh confident sentence.

Frequently Asked Questions

Why does AI sound confident when it's wrong? Because the confident tone is a style it learned, not a measure of accuracy. Language models were trained on finished, edited prose written by people who knew the answer, so they inherited that assured register. Researchers led by Miao Xiong found that models "tend to be overconfident" when they state confidence, "potentially imitating human patterns of expressing confidence." ¹ The voice is copied; it is not evidence.

Can I trust how confident an AI sounds about my notes? No. Stated confidence is systematically too high and only loosely tied to whether the answer is right. Xiong's ICLR 2024 benchmark measured this across five dataset types and five models, GPT-4 included, and found verbalized confidence overconfident throughout. ¹ Treat a confident answer about your notes as a claim to verify against the file, not a signal that the model checked and was correct.

Are LLMs overconfident about their answers? Yes, when they verbalize confidence. The finding held "across five types of datasets (e.g., commonsense and arithmetic reasoning) and five widely-used LLMs including GPT-4 and LLaMA 2 Chat" ¹ — so it is not one odd model. Even reading a model's internal probabilities instead of its words barely helps: the failure-prediction AUROC gap was "narrow, e.g., 0.522 to 0.605," close to a coin flip. ¹

Does an AI know when it's wrong? Partly. An Anthropic preprint found "larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format," ³ so some self-knowledge exists. But the same work notes models "struggle with calibration of P(IK) on new tasks" ³ — precisely the unfamiliar terrain your personal notes occupy. The partial signal is real and not enough to lean on.

Does asking the AI for its confidence make it reliable? It helps, but does not make it safe. Stanford researchers found that asking for verbalized confidence cut expected calibration error by "a relative 50%" on several benchmarks. ⁴ That is a large improvement measured down from a poorly-calibrated baseline, not a fix — and Xiong's team found "all investigated methods struggle in challenging tasks." ¹ Asking narrows the gap; it does not remove the need to verify.

How do I check if an AI answer about my notes is accurate? Make the claim mechanical instead of trusting the tone. Ask the model for the verbatim sentence it is relying on, then search your files for it with rg -F "the quoted sentence" vault/. If the span is not there, the answer is unsupported regardless of how sure it sounded. Because notes stored on your own device are plain text, that check is local and takes one command.

The old habit was to read a fluent answer and feel reassured. The new habit is to read the same answer and ask it to prove itself against the file, because a model's certainty is a register it copied, not a fact it verified — and the only thing that settles the question is the note itself.

MNMNOTE keeps your notes as plain Markdown on your own device, so any confident claim an AI makes about them is one local search away from confirmed or falsified — mnmnote.com.