The Same Prompt Stopped Working: Write Down Which Model Made It

When the same prompt that worked last month returns something worse today, the cause is often not your prompt. The model changed under you. You can't pin a hosted model in place, but you can record the one thing you own: which model snapshot, on what date, with which settings made a result, in a one-line note beside the output.

That habit is the personal-scale version of a practice teams already formalize. A December 2025 software-engineering study that catalogs recurring failure patterns in AI code names exactly this gap as a "code smell" called No Model Version Pinning, and prescribes the fix in one sentence: "Always specify an immutable identifier and record it with other run metadata." ¹ This is a how-to for doing that without a registry, a tracker, or an account — just a note next to the answer.

Why the same prompt stops working

A hosted model behind a plain name like gpt-4o or claude-sonnet-4-5 is a moving target. The name is a pointer, not a fixed thing. When the provider ships a newer dated snapshot, the pointer can quietly move to it — and your "same prompt" is now talking to a different model. The output drifts, and nothing in your text changed.

OpenAI documents the default state plainly: "The Chat Completions and Completions APIs are non-deterministic by default (which means model outputs may differ from request to request)." ² Even when you fix everything you can, the floor is not stable. A 13-author study found "accuracy variations up to 15% across naturally occurring runs with a gap of best possible performance to worst possible performance up to 70%" on settings meant to be deterministic. ³

There is also a layer you can't see at all. OpenAI exposes a system_fingerprint: "an identifier for the current combination of model weights, infrastructure, and other configuration options used by OpenAI servers to generate the completion." ⁴ It "changes whenever you change request parameters, or OpenAI updates numerical configuration of the infrastructure serving our models" — which, the docs note, "may happen a few times a year." ⁴ A few times a year, the thing answering you shifts. You only know it happened if you wrote down what you had before.

The alias is not the snapshot

The core distinction is between an alias and a snapshot. An alias is a convenience name pointing at "whatever the latest version is." A snapshot is a specific, dated version that does not move. Anthropic states it directly: an alias such as claude-sonnet-4-5 "is a convenience pointer that resolves to the most recent dated snapshot for that minor version." ⁵

The same name can therefore mean two different models a month apart. A snapshot is the opposite. OpenAI's docs: "Snapshots let you lock in a specific version of the model so that performance and behavior remain consistent." ⁶ Anthropic's pinned model IDs work the same way: "Each Claude model ID identifies a pinned version of the model. When you use a model ID in an API request, the underlying model remains constant for the lifetime of that ID." ⁷ The study puts the cost of skipping this in one line: "Using only a provider alias removes explicit versioning, so weights, prompts, and safety filters can change without notice and shift behavior." ¹

So the first move is to call the dated snapshot, not the bare alias, whenever a result matters. The second move — the one most people skip — is to record which snapshot you called, next to the output it produced.

The one-line provenance note

The fix is small enough to be permanent. When a result matters enough that you'd hate to lose its lineage, write one line next to it: the model snapshot, the date, and the key settings. That line is your provenance note. It travels with the output and answers the question that matters later — "what produced this?"

A provenance note has three required fields and an optional fourth:

• Model snapshot — the dated, immutable ID, not the moving alias. gpt-4o-2024-11-20, not gpt-4o.
• Date — when you ran it, so a later "it changed" has a before.
• Key settings — temperature, and any system prompt or top-p you set deliberately.
• Fingerprint (optional) — if the API returns a system_fingerprint, paste it. It's the silent layer made visible.

Here is the whole method as a copy-pasteable block. Drop it under any AI-assisted result you'd want to reproduce or audit later:

## Provenance — draft intro, v3

- **Output:** [link or paste the result this note describes]
- **Model:** gpt-4o-2024-11-20        # the dated snapshot, not `gpt-4o`
- **Date:** 2026-06-23
- **Settings:** temperature 0.2 · top_p 1 · system prompt: "house editorial voice"
- **system_fingerprint:** fp_a1b2c3d4   # optional; the layer you can't otherwise see
- **Why it mattered:** approved opener — need to reproduce the tone for the series

# Cross-vendor example, same shape:
# - **Model:** claude-sonnet-4-5-20250929   (a pinned ID, not the `claude-sonnet-4-5` alias)

Why this exact shape, field by field. The snapshot is the thing that moves — naming the dated version is what makes "which model" answerable at all. The date turns a vague "it used to be better" into a checkable claim against a changelog. Settings catch the case where you moved, not the model. The fingerprint is the only visible trace of the infrastructure layer the provider can change "a few times a year." ⁴ Omit any field and you reintroduce the exact ambiguity the note exists to remove.

How a provenance note diagnoses drift

The note's payoff comes when an answer changes and you need to know who moved. With a provenance note beside the old output, the question turns mechanical: compare what you recorded against what you run now. If the snapshot or fingerprint differs, the model moved. If they match but your settings differ, you moved. If everything matches, it's irreducible non-determinism.

flowchart TD
  A[Answer changed] --> B{Snapshot or<br/>fingerprint differ?}
  B -->|Yes| C[The model moved]
  B -->|No| D{Your settings<br/>differ?}
  D -->|Yes| E[You moved]
  D -->|No| F[Irreducible<br/>non-determinism]

Figure: How a provenance note diagnoses drift. When an answer changes, you first check whether the recorded model snapshot or fingerprint differs from what you're running now — if so, the model moved. If they match, you check your own settings — a difference there means you moved. If both match and it still drifted, you're seeing the run-to-run variation the vendors document. Without the recorded note, none of these three causes can be told apart.

This is also where the note earns its place alongside two companion habits: a decision log of what you asked the AI and why, and a plain-text diff of two AI answers. The decision log records what you asked. The diff shows what changed. The provenance note records what produced each answer — so when the diff reveals drift, you can tell whether the model moved or you did. The three are a set: a log without provenance can't be reproduced, and a diff without provenance can't be diagnosed.

What the note does and does not do

Be honest about the limits, because overselling this habit is the fastest way to lose trust in it. A provenance note improves reproducibility and auditability. It does not guarantee identical output, and it does not judge whether the output is correct. Four caveats keep it honest.

• Pinning improves reproducibility; it does not guarantee it. Even a fixed model ID can shift: "Occasionally, infrastructure updates produce minor differences in observable behavior even when the model ID and weights have not changed." ⁷ And a seed is, in OpenAI's words, only "best effort" — "Determinism is not guaranteed." ⁸ The note's real value is knowing what to blame, not certainty.
• Bit-for-bit determinism is realistic only on open weights you run yourself. A fixed seed, temperature zero, and a controlled backend you operate can get close. For hosted models, you don't hold the backend — so the provenance note is the practical fix, not full determinism.
• The note records what produced a result, not whether it's right. Like a diff, it is diagnostic. Correctness is still your judgment.
• It does not make an answer safe to act on. Recording which model wrote a medical, legal, or financial answer does not make that answer correct. For regulated decisions, a model's output is not a substitute for professional advice.

How this fits a plain-text workflow

The reason to write the note in plain text, in a file you own, is durability. Steph Ango, Obsidian's CEO, frames the principle well: "Apps are ephemeral, but your files have a chance to last." ⁹ A note in a registry dies with the registry. A note in a Markdown file survives every tool you switch to, because it's just text.

That is the whole architecture of the method: a note you own, on your own device, sitting beside the result it explains. No model registry, no experiment tracker, no account — those are the team-scale tools, and they're a fine choice when you have a team. For one person who just wants a result they can reproduce or audit a month later, the answer is a line of Markdown. MNMNOTE keeps that note in plain text, local-first, beside whatever you wrote. The point is not the tool; the point is that the note outlives the model that made the result.

Frequently Asked Questions

Why does ChatGPT give different answers when asked the exact same question multiple times? Two causes overlap. Hosted models sample probabilistically, so output "may differ from request to request" by default. ² Separately, the model itself can change — a moving alias gets repointed to a new dated snapshot, or the system_fingerprint shifts. ⁴ A provenance note lets you tell which cause is in play.

Why did my prompt stop working when I didn't change anything? Because a plain model name like gpt-4o is a pointer to "whatever's latest," not a fixed model. When the provider ships a newer snapshot, the pointer can move and your prompt now meets a different model. ⁵ Calling the dated snapshot and recording it next to the output is what catches this.

Does temperature 0 guarantee deterministic LLM output? No. OpenAI's own docs say "Determinism is not guaranteed," and a seed is only a "best effort." ⁸ A 13-author study measured "accuracy variations up to 15%" even on settings meant to be deterministic. ³ You record provenance precisely because settings alone don't fully pin a hosted model.

How do I make an AI-assisted result reproducible? On a hosted model you can't make it bit-for-bit reproducible, but you can make it auditable: record the model snapshot, the date, and the key settings next to the output. The cited prescription is "specify an immutable identifier and record it with other run metadata." ¹ That record is what reproducibility means at personal scale.

What is the difference between gpt-4o and gpt-4o-2024-11-20? The first is an alias — a moving pointer to the latest version of that model. The second is a dated snapshot that does not move: "Snapshots let you lock in a specific version of the model so that performance and behavior remain consistent." ⁶ Call and record the dated one when a result matters.

Does pinning the model version make the output identical every time? No. Pinning improves reproducibility and auditability; it doesn't guarantee identical output. Even a fixed ID can drift when "infrastructure updates produce minor differences in observable behavior." ⁷ The note's value is knowing what to blame when an answer changes — not certainty.

Do I need an MLOps platform or experiment tracker to do this? Not as an individual. Registries and trackers are the team-scale version of "record it with run metadata." ¹ At personal scale, a one-line provenance note in a plain-text file you own does the same job: it tells you which model and settings produced a given saved output.

A hosted model is the one part of your AI workflow you will never control. The note that records which one made a result is the part you always can — so write it down, next to the answer, in a file that will outlast the model.

This builds on the academic prescription to "specify an immutable identifier and record it with other run metadata" ¹ — to keep that record in plain text you own, mnmnote.com sits beside whatever you wrote.