Got a Different AI Answer? Diff Them

When you re-ask an AI the same question and the answer comes back different, paste both answers into one note and run a text diff. The diff shows exactly which lines changed — added, removed, reworded. No eval platform, no dashboard. An AI answer is just text, so two answers are just two files.

This works because the change is real and the cause is not your imagination. Hosted models are non-deterministic by design, even at temperature=0. The fix is not a setting you can flip. It is a habit you can keep: save both answers, diff them, read the change. One honest caveat up front, repeated below — a diff tells you what changed, never which version is correct.

Why the same question gives different answers

The model is non-deterministic by default, so identical prompts can return different text. OpenAI's own documentation states it plainly: "The Chat Completions and Completions APIs are non-deterministic by default" ¹. This is not a bug you triggered. It is the standard behavior of a hosted endpoint, and it holds even when every word of your prompt stays the same.

People assume temperature=0 removes the randomness. A 13-author study found the assumption is widespread: "Many users of LLM platforms expect model performance to be deterministic when temperature=0" ². The expectation is reasonable and wrong. Temperature controls one source of variation, not all of them.

Why temperature 0 does not fix it

Setting temperature=0 narrows the sampling but does not make a hosted model repeatable. Shuveb Hussain of Unstract puts it directly: "even with temperature set to 0, LLM responses can still be non-deterministic due to several technical factors that most of us don't think about until they bite us in production" ³. The randomness is one layer down.

Two mechanisms do most of the damage. The first is arithmetic. GPUs and CPUs "use floating-point math that isn't associative: (a+b)+c can differ from a+(b+c) by a few ULPs (units in the last place)" ³. Reorder the additions and the result shifts in the last decimal places — enough to change which token wins.

The second mechanism is the server you share. Horace He of Thinking Machines Lab traced model non-determinism to "batch invariance" — your output depends on how many other requests were batched alongside yours: "it's because our forward pass lacks 'batch invariance', causing our request's output to depend on the batch size of our forward pass" ⁴. Server load decides the batch size, and load is not under your control.

He measured the effect. At temperature 0, 1,000 completions of one prompt produced "80 unique completions, with the most common of these occuring 78 times" ⁵ — same prompt, same settings, eighty different answers. And it is not a GPU quirk — "This nondeterminism is not unique to GPUs — LLM inference endpoints served from CPUs or TPUs will also have this source of nondeterminism" ⁶.

What about a seed? It helps and it still does not promise repeatability. OpenAI's seed guidance is explicit: "If specified, our system will make a best effort to sample deterministically" — then the warning, "Determinism is not guaranteed, and you should refer to the system_fingerprint response parameter to monitor changes in the backend" ⁷. Best effort is not a guarantee.

Why a diff is the right tool

A text diff answers the exact question you have — "what changed?" — without pretending to answer a harder one. The academic finding makes the case for needing it: across runs, "none of the LLMs consistently delivers repeatable accuracy across all tasks, much less identical output strings" ⁸. If the strings differ, comparing strings is the honest first move.

The reason eyeballing a diff beats a strict equality check is that two answers can mean the same thing while reading differently. Future AGI's guidance is to "evaluate with semantic equivalence rather than byte-exact match" ⁹. A diff supports exactly that judgment: it surfaces the changed lines and leaves the meaning call to you, instead of failing loudly on a comma.

When to capture the second answer

Capture answer A the first time an AI answer matters enough to act on, and answer B at the moments a hosted model is likely to have shifted underneath you. You cannot diff what you did not keep, so save early — then re-ask at the points where drift is plausible, not on a paranoid loop.

Three moments are worth a re-ask. The first is a vendor model update — the seed documentation tells you to "refer to the system_fingerprint response parameter to monitor changes in the backend" ⁷, so a changed fingerprint is a direct cue to diff. The second is when a once-reliable prompt suddenly feels off; the change can be the model, not you.

The third is a slow calendar check on prompts you depend on, because the 13-author study found drift "across naturally occurring runs" at fixed settings ², not only after edits. Between those moments, one saved baseline is enough.

The five-minute version

The fast version is four steps and needs nothing but a note and your eyes. Save answer A, get answer B, put them side by side, and read the difference. You are comparing two files, because that is all the answers are.

1. Keep answer A. The first time an AI answer matters, paste it into a note. Date it. This is your baseline — the thing you will compare against later.
2. Get answer B. Re-ask the same prompt, or ask a second model, and paste that answer into the same note under a heading like ## B.
3. Read them side by side. With both blocks in one note, scan for what moved: a dropped caveat, a changed number, a reordered list, a new claim. Most drift is visible to the naked eye in under a minute.
4. Write one line on what changed. "B dropped the rate-limit warning" or "B changed the date from 2024 to 2025." That sentence is the whole point — it is the answer to "what actually changed?"

The thirty-minute version

The longer version makes the comparison exact instead of eyeballed: save each answer as a file and run a real diff. The tool is built for it — git diff is documented as "git-diff - Show changes between commits, commit and working tree, etc" ¹⁰. The POSIX diff does the same line-by-line comparison, in plain text you keep.

1. Save each answer as a .md file. answer-a.md and answer-b.md, one paragraph or list item per line so the diff stays readable.
2. Run the diff. diff answer-a.md answer-b.md prints the changed lines. For word-level changes inside a line, git diff --no-index --word-diff answer-a.md answer-b.md shows additions and deletions inline.
3. Read the markers. Lines prefixed with what was removed and what was added are the change. Everything unmarked is identical between the two answers.
4. Commit the pair if it matters. git init a folder of prompts and answers, and each git diff against the last commit becomes a record of how the model's answer to your question drifted over time — history you own, on your own device.
5. Graduate when the one-off keeps happening. If you find yourself diffing the same prompt every week, that is the signal to write a reusable fixture instead. That is a different habit, covered in the companion piece on plain-text eval files.

Here is what a real diff looks like. You asked twice for a refund-policy summary; the second answer quietly dropped a window and softened a guarantee. Running git diff --no-index answer-a.md answer-b.md prints exactly that:

 Refunds are processed to the original payment method.
-Eligible within 30 days of delivery.
+Eligible within 14 days of delivery.
-Shipping costs are fully refunded.
+Shipping costs may be refunded at our discretion.

The lines marked with a leading minus are what the first answer said; the plus lines are the second answer. Unchanged lines are not shown. In five seconds you can see the model moved the refund window from 30 days to 14 and turned a flat guarantee into a discretionary one. Whether 14 or 30 is correct is your call — the diff only tells you the number moved.

Common mistakes

The diff habit fails in five predictable ways, and each one is avoidable. Most failures come from treating the diff as more than it is — a correctness verdict, a determinism fix — or from keeping too little to compare against in the first place. Read the list once before you start, and the habit holds.

• Keeping nothing to compare against. You cannot diff one answer. The discipline is to save answer A before you need it — the moment an AI answer matters, not after it has already changed. No baseline, no diff.
• Reading the diff as a verdict on correctness. A diff shows wording and structure that changed. It does not tell you which version is right. That judgment stays yours — the controlled-AI discipline is about owning that call, not outsourcing it.
• Expecting a setting to stop the drift. Do not promise yourself that temperature=0 or a seed will make the model repeat. On a hosted endpoint they will not. Bit-for-bit reproducibility is realistic only on open weights you run yourself with deterministic, batch-invariant kernels ⁴ — not in the hosted world this habit is built for.
• Diffing two answers that meant the same thing. A reworded sentence is not necessarily a regression. Use the diff to find the changed lines, then judge meaning yourself; do not fail an answer because a comma moved.
• Acting on either answer in a regulated domain. Comparing two AI answers does not make either one correct or safe to rely on. For medical, legal, or financial decisions, a diff is not a substitute for professional advice.

Keeping both answers as files you own

Because an AI answer is just text, the whole habit lives wherever you keep plain text. The point of saving both answers is ownership — you hold the baseline, the comparison, and the record of what changed. Steph Ango, CEO of Obsidian, frames the durable version: "Apps are ephemeral, but your files have a chance to last" ¹¹.

Two answers kept as two files outlast whichever model produced them. The model can change underneath you tomorrow; the files do not, and neither does your ability to read exactly how the answer drifted.

A local-first markdown note is enough for the whole flow. Paste answer A and answer B into one note, eyeball the difference, or export each to a .md file and run a real diff in your terminal — all offline, no account, nothing leaving your machine. The method is tool-agnostic: any editor plus any terminal works. What matters is that you kept something to compare against, in a format that is still readable long after the answer changed again.

Frequently asked questions

The short version: hosted models are non-deterministic by default, temperature=0 does not fix it, and a text diff is the fastest honest way to see what changed between two answers. The questions below cover why the drift happens, how to run the comparison, and the one thing a diff deliberately does not tell you.

Why does ChatGPT give different answers to the same question every time?

Because hosted models are non-deterministic by default — identical prompts can return different text ¹. The causes are probabilistic decoding plus hardware effects: floating-point math that is not associative ³ and "batch invariance," where your output depends on how many other requests were batched with yours ⁴. Even temperature=0 does not remove this ³.

Does temperature 0 guarantee deterministic LLM outputs?

No. OpenAI's documentation says "Determinism is not guaranteed" even with a seed set ⁷. A 13-author study measured "accuracy variations up to 15% across naturally occurring runs" at fixed settings ². Temperature controls one source of variation; server-side batching and floating-point order introduce more ^{3 4}.

How do I compare two AI answers and check if the output changed?

Paste both answers into one note and read them side by side, or save each as a .md file and run diff answer-a.md answer-b.md. For word-level changes, git diff --no-index --word-diff shows additions and deletions inline ¹⁰. The diff prints exactly which lines changed.

Does a text diff tell me which AI answer is correct?

No. A diff shows what wording and structure changed, not which version is right. That judgment stays yours — comparing the answers tells you where they disagree, then you decide which one to trust. For regulated domains, neither answer is automatically safe to act on.

Why is deterministic output from LLMs nearly impossible?

Because the math and the infrastructure both introduce variation. Floating-point addition is not associative, so reordering changes the result by a few units in the last place ³, and batch size varies with server load, so "the load (and thus batch-size) nondeterministically varies" ⁶. Both shift which token the model picks.

Do I need an eval platform to catch when my AI output drifts?

Not for an ad-hoc check. Two files plus diff in a note you own answers "what changed?" without any platform. When you find yourself running the same comparison on a schedule, that is the moment to graduate to a reusable plain-text eval file — the repeatable version of this one-off reflex.

A re-asked answer is not a mystery and not a betrayal; it is a second file, and the difference between two files is something you can read in a minute. Keep both, diff them, and the question "what actually changed?" stops being a shrug and becomes a line you can point to. To keep both answers as plain text you own, mnmnote.com lives in your browser.