Ask the AI the Same Question Twice, Get Two Answers

Ask an AI the same question about your notes today and tomorrow, and you can get two different answers — even with temperature set to 0, the setting that is supposed to make it repeat itself. The reason is not in your prompt. It is on a server you don't control. So save the answer.

This is not the folklore that "AI is random," and it is not a bug in any one model. It is a measured property of how hosted models are served. In September 2025, researchers at Thinking Machines Lab ran the same prompt one thousand times at temperature 0 against a single open model and got 80 distinct completions, the most common appearing only 78 times.¹ Temperature 0 is meant to remove the dice, to make the model pick its single most likely next word every time. It did not. "Even when we adjust the temperature down to 0," the team wrote, "LLM APIs are still not deterministic in practice."¹ The variance is small. But small and invisible is exactly the wrong combination for a note you intend to rely on.

What temperature 0 was supposed to fix

The common advice is sensible and almost right: if you want the same answer twice, set the temperature to 0. Temperature controls how much randomness the model adds when it picks each word. Turn it to 0 and, in theory, the model always takes its single most likely path. Same input, same output.

The belief is so reasonable that careful people hold it. Simon Willison, who has documented language-model behavior for years, admitted he had been "lead to believe this was due to the non-associative nature of floating point arithmetic"² — the idea that tiny rounding differences, summed in a different order, nudge the result. It is a good theory, and the math under it is real: add the same floating-point numbers in a different order and you can land on a slightly different total. That much is true. It is simply not the main reason your notes answer changes from one run to the next. The Thinking Machines team called the popular "concurrency + floating point" hypothesis a story that "misses the mark," and went looking for the true culprit.¹

Why the same setup still gives you two answers

The real cause sits on the server, not in your settings. When you send a prompt, it joins other users' prompts in a batch, and the batch size shifts from moment to moment with load. That changing batch size changes the order of certain calculations, which changes the output — the same prompt, a different crowd.

Thinking Machines Lab traced it precisely, and stated it plainly: "the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies."¹ You cannot see this batch. You cannot set it. It is decided by how many other people happen to be talking to the same model at the same instant — a number that has nothing to do with you and everything to do with your answer. In their thousand runs, the completions began identically and only split apart at the 103rd token.¹ The answer holds, word for word, and then quietly forks. That is the unsettling part: the divergence hides deep in an answer that starts out looking stable.

Determinism is real — it just isn't yours to switch on

This does not mean an AI can never repeat itself. Engineers have built serving systems that return bitwise-identical answers every time; the variance is a choice, not a law of physics. The catch is that the choice is made on the server, and it costs speed. It is almost never the default, and it is not a switch you hold.

When the Thinking Machines team turned on what they call batch-invariant kernels, written so the math comes out the same regardless of batch size, the result was blunt: "all of our 1000 completions are identical."¹ A separate group, working on the SGLang serving system, reached the same destination and published the price. In a September 2025 write-up they reported "fully deterministic inference"³ on their stack, bought by trimming the performance cost to 34.35% overhead, down from an earlier 61.5%, with most slowdowns "ranging from 25% to 45%."³ Those figures are specific to one system at one moment. Read them as an illustration, not a constant. The durable lesson survives the specifics: repeatability exists, and it is paid for in throughput that most providers would rather spend on being fast. So the default you meet is the fast one, not the reproducible one.

What a seed can and can't do

Some providers hand you a partial lever: a seed. Pass the same seed and prompt, and the system will, in OpenAI's words, "make a best effort to sample deterministically." Best effort is the operative phrase. A seed reduces variance within one backend; it does not guarantee a repeat, and it breaks the moment the provider changes anything underneath you.

OpenAI is candid about the limit. Repeated requests with the same seed "should return the same result," the documentation says, and then, in the same breath, "Determinism is not guaranteed."⁴ To help you notice when the ground has shifted, the API returns a system_fingerprint, an identifier for the current backend configuration; when that fingerprint changes, the same seed can produce a different answer, and you are meant to treat the two as no longer comparable.⁴ So the seed is genuinely useful and genuinely not a guarantee. It narrows the odds. It does not close them. And it says nothing about which model version answered you in the first place — a separate provenance question, because hosted weights change without announcement too, and different weights answer differently.

What to save

Since you cannot make the answer reproducible, save it instead. Treat every AI answer about your notes as a one-time event: capture the output and the inputs that produced it — the prompt, the model, the date — in a plain-text note you own. What you can't re-derive, you keep. What actually matters, you recompute with a real tool.

• Save the answer the first time. If an AI's summary or judgment about your notes matters, paste it into a note the moment you get it. Re-asking is not a backup — the second answer may not match the first, and neither one is on file unless you put it there.
• Record what produced it. One line beside the output, naming the prompt, the model and its version, and the date, turns a disappearing reply into a durable record. The version earns its place because different weights answer differently, and hosted models change without telling you.
• Recompute what's load-bearing. For anything you would act on — a total, a count, a deadline — don't trust the generated sentence; recompute it with a real tool and save that result next to the answer. A number you can reproduce beats a sentence you can't.
• Stop expecting a rerun to match. Ask twice and you may get two plausible answers. That is the system working as built, not a fault to debug. Keep the one you trusted, and write down why you trusted it.

Frequently asked questions

These are the questions people type after a chatbot answers the same thing two different ways, answered directly. The short version runs through all of them: temperature 0 does not promise a repeat, the cause is server-side batching you can't reach, and the fix is not to chase reproducibility but to save the answer you got.

Does setting temperature to 0 make the output deterministic?

Not reliably. Temperature 0 makes the model prefer its single most likely next word, but the hosted system around it still varies. Thinking Machines Lab ran one prompt a thousand times at temperature 0 and got 80 different completions.¹ Lower temperature narrows the variety; it does not promise an identical answer twice.

Why does ChatGPT give you a different answer every time?

Mostly because of how it is served, not because it is "random." Your request is processed in a batch alongside other users' requests, and that batch changes size with traffic, which shifts the underlying math just enough to change the words.¹ Your prompt, your settings, and the model can be identical, and the answer still moves.

Does ChatGPT give the same answers to everyone?

No, and not even to the same person twice. Sampling adds variation, your conversation history feeds different context, and server-side batching nudges the output run to run.¹ Two people asking the identical question, or one person asking it twice, can each receive a slightly different, equally confident reply.

How do I make the AI's output reproducible?

From the client, you mostly can't. Some APIs offer a seed that makes a "best effort" to repeat, but "Determinism is not guaranteed," and it breaks when the provider updates the backend.⁴ Full reproducibility is a server-side feature that costs speed,³ so most providers leave it off. Save the answer instead of re-deriving it.

Why does ChatGPT give different answers in the same session?

Even back to back, the computation underneath is not pinned. The model samples with some randomness, and the serving system batches your two requests differently depending on load, so identical questions can diverge.¹ The first answer is not more correct than the second — which is exactly why the one you rely on belongs in a saved note.

If I can't reproduce the answer, should I still use AI on my notes?

Yes — the takeaway is a habit, not a reason to stop. An AI is useful for reading and summarizing your notes; it is simply not a stable source of record. Use it, then save its output and the inputs that made it, and recompute anything load-bearing with a real tool.³ Treat the answer as a draft you keep, not a fact you can summon again.

An answer you can't reproduce is not a fact you can look up again — it is a moment that happened once. The model will not hand you the same words twice, because the sameness was never yours to command. So the discipline is quiet and old: when the machine says something worth keeping, keep it, in a place that will still be yours after the model has moved on.

Because a note in an open Markdown file lives on your own device, the answer you save today reads exactly the same tomorrow, whatever the model does next, in mnmnote.com.