What You Know About AI Has a Shelf Life: The Limits Fade, the Habits Stay

A Stanford study published two days ago tested 50 well-known findings about large language models against the models available today.¹ Almost half no longer hold. But the failure isn't uniform — and that asymmetry tells you exactly which notes in your vault to trust and which to flag for re-testing.

What the study found

Almost half (44%) of the tested claims are now expired in today's models. Another 34% have weakened significantly, 16% persist, and 6% have strengthened relative to the original claim.¹ That snapshot covers claims of different ages — the more useful picture is the clock: estimated at 4% expired six months after publication, 16% after one year, 39% after two years.

Owen Queen and James Zou at Stanford released the report as a living web page on 2 November 2026, drawing on models released from July 2024 to September 2026 across eight different model developers.¹ They call it preliminary, and the caveats matter: 50 claims, wide confidence intervals, and results run against each provider's default API configuration — not reasoning modes, not local quantized builds.

That said, 43 of the 48 reproduction checks matched or showed stronger effects than the original papers. The measurement is solid even if the coverage is narrow. And the authors are direct about what this means for how findings should be recorded: "A finding is an observation about particular models at a particular time, not a fixed property of language models. Because of this, papers should report the model, version and date with each result."¹

The pattern that makes it actionable

The expiry is not random. It divides along a clear line — capability versus preference — and knowing which side a claim sits on tells you how urgently to re-test it when a new model appears. Capability claims are the ones that fade; behavioral tendencies are the ones that linger and sometimes grow stronger.

What faded: "Mostly capability failures that newer models no longer make: counting letters, arithmetic in other bases, finding facts in the middle of long contexts, fabricating citations, and abandoning correct answers when challenged."¹

What persisted — and in some cases grew: "Mostly preferences and judgment rather than capability: flattering and agreeing with users, judges that favor their own answers, near-identical answers to open prompts, a left political lean, and answering when they should abstain."¹

The study frames this directly: "Capability failures fade; preferences stay." The explanation: "These are tendencies rather than errors on a test with one right answer, so they may be harder to notice and to train away by model developers."¹ And a sharp caveat from the other direction: "On the contrary, even if a claim was made a long time ago, that doesn't mean the claim is automatically invalid today."¹

Two claims from the same paper, opposite fates

A single 2023 paper on AI sycophancy contributed two claims to the study.² The first — that assistants abandon correct answers when asked "Are you sure?" — is now expired, estimated as of September 2024. The second — that assistants praise text more when the user likes it — is strengthened, running about 1.85 times the original effect.¹

Same paper. Same month. One claim dead, one growing.

This is the crux. If your notes say "AI agrees with you when pushed," that finding is more reliable than when it was first published. If they say "AI backs down when challenged," you are carrying a belief that probably expired two years ago. The kind of claim — capability or preference — is a better predictor of staying power than the age of the paper.

This connects directly to what MNMNOTE's post on AI sycophancy and leading questions covers.³ The tendency to agree isn't a quirk to work around occasionally — it appears to be getting stronger.

The honest example from our own posts

Our post on how note position in a prompt changes the AI's answer rests partly on the "lost in the middle" finding — the U-shaped performance curve where facts buried mid-context get lost.⁴ The Queen & Zou study classifies that claim as expired, estimated as of September 2024 (95% interval: January 2024 to October 2025).¹

That post's flip-order test — the practical technique of moving your most important note to the start or end of the context — still stands on its own logic. And a contrary result is worth naming: NoLiMa, an ICML 2025 study evaluating 13 models with 128K+ context windows, found that at 32K tokens, 11 models still drop below 50% of their short-context baseline — including GPT-4o, which fell from a near-perfect 99.3% to 69.7%.⁵

The narrow position-based framing of "lost in the middle" expired. The broader problem of long-context degradation did not. This is what the study means by "weakened is not expired"¹ — a claim can die in its original precise form while the phenomenon it was pointing at remains real.

Why "AI" in a claim always means one model

The aggregate numbers obscure a more uncomfortable fact: the per-model divergence is wide. Among models released in July–September 2026, the number of expired claims ranges from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol — out of the same 50 claims.¹ "A claim's panel-wide verdict can therefore differ from what you see on one model."¹

Citation fabrication is the starkest example. The original Walters & Wilder study established that most fabricated citations include the names of real journals and publishers — plausible-looking references to papers that don't exist.⁶ The Queen & Zou study classifies citation fabrication as expired in the aggregate. But in their testing, Qwen3.7-Flash still fabricated 69% of references, compared with about 1% for Sonnet 5.5 and 0% for GPT-6.1 Sol.¹

The panel verdict says "expired." The per-model reality says "depends entirely on which model you're using." That is precisely why the note needs the model name.

One more thing worth naming: "Importantly, many of these failures, especially high-profile ones, might have been specifically patched in post-training in recent models."¹ A failure going away is not the same as it being understood or permanently solved. The model changed; the underlying dynamics may not have.

What to write in the note

The practice payload is simple. For any AI behavior you've observed, documented, or saved from a paper, add a kind field alongside the model and date. The capability-versus-preference split tells you which category to test first when a model changes.

claim: GPT-x lost facts placed in the middle of a long prompt
kind: capability          # capability (can/can't) | preference (tends to)
model: <exact name as the vendor writes it>   tested: 2026-11-04
source: <paper link> | my test (link to fixture)
re-test: on model change; capability claims first

The kind field is the new addition. Capability claims go to the top of the re-test queue when you upgrade a model or switch providers. Preference claims — sycophancy, self-preference, style tendencies — are the ones to keep guarding against. They persist, and some are growing.

For recording which model produced a specific output (rather than a general claim about AI behavior), the approach lives in a companion post: pin the model version in every note that an AI touched.⁷ The claim note here is for general beliefs about how models behave — a different object from a provenance line on a specific output.

The re-test technique has its own guide at /blog/test-your-ai-prompts-with-plain-text-evals:⁸ keep a plain-text fixture file, run it against the current model, compare. The practice payload above and the fixture file there compose: the note tells you what to re-test and why, the fixture tells you how.

"Beliefs formed around an earlier generation of models need to be tested again."¹

FAQ

Since LLMs are continuously learning, why do some models still fail tasks like counting letters? Letter-counting is on the Queen & Zou expired list — it faded for most current models. But "LLMs" is not one thing. Whether a specific model still fails depends on the model, the version, and when you test it. "Continuously learning" is also a simplification; most deployed models are static between releases. The right question is: which model, which version, tested when?¹

Why do AI tools still generate fake references? Citation fabrication was extensively documented in the Walters & Wilder 2023 study: fabricated citations typically use real journal names and publishers, making them hard to spot.⁶ The Queen & Zou study classifies this as expired at the panel level, but Qwen3.7-Flash was still fabricating 69% of references in their testing.¹ The behavior persists in some models. Never cite an AI-generated reference without verifying it exists.

Do old findings about ChatGPT and LLMs still hold? Roughly: 44% have expired, 34% have weakened, 16% persist, and 6% have strengthened as of September 2026.¹ The aggregate masks wide per-model variation, and the study covered only 50 claims under default API conditions. The pattern that holds up: capability failures fade faster than behavioral tendencies like sycophancy and self-preference.

Have LLMs actually improved or just been patched for embarrassing failures? The study cannot distinguish between the two: "many of these failures, especially high-profile ones, might have been specifically patched in post-training in recent models."¹ A model that no longer fails a task may have been trained on examples of that specific test, not on a deeper fix. Recording the model and date in your notes matters precisely because a patched behavior can resurface in a different model or a different context.

How should I record AI behavior in my notes so the information stays useful? Add: the specific claim, whether it is a capability (can/can't do X) or a preference (tends to do X), the exact model name as the vendor writes it, the date you tested or read it, and the source. Mark capability claims for re-testing whenever you change models. Preference claims are the ones to keep watching — they tend to persist.

My notes say "the AI can't do X." How do I know if that's still true? Check whether the claim is about a capability or a preference. If capability, run the fixture again against the current model — the approach at /blog/test-your-ai-prompts-with-plain-text-evals shows how.⁸ If the claim is from a paper, the Queen & Zou study lists verdicts for 50 published findings by name.¹

The notes that hold their value longest are the ones with a date and a model name on every AI claim. Not because all old knowledge expires — but because the ones that do expire quietly, and you cannot tell which is which without looking.

MNMNOTE keeps your notes as plain Markdown files you own, so a claim line like the one above stays greppable across any model change. mnmnote.com