Your Local Model Is a Different Model

The model you run on your own machine is not the model the benchmarks measured. It is a compressed build of it — and compression changes individual answers far more than it changes the score. In one measured configuration, accuracy moved 2.36 points while 13.62% of the answers flipped outright.

Running models locally is no longer a fringe position. "Local AI needs to be the norm" drew 1,903 points and 749 comments on Hacker News in May 2026.¹ Yet the discussion treats run it locally as one decision. It is two: which model, and which build of that model.

Four researchers at Microsoft Research Bangalore measured what the second decision costs. Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra and Ramachandran Ramjee published Accuracy is Not All You Need at NeurIPS 2024, and their conclusion fits in one sentence: "We show that accuracy severely underestimates the true distance between models as perceived by the end user."²

Underestimates by how much is the interesting part.

The score says it is the same model

Quantization shrinks a model by storing its weights at lower precision, and the published benchmark scores barely move. Across the six schemes in the NeurIPS study's headline comparison, accuracy sat within 0–2% of the 16-bit original.² That number is the whole basis of the practitioner shorthand: the small build is fine, ship it.

The shorthand is not wrong about size. llama.cpp's own documentation is blunt about the trade it offers: "Quantization reduces the precision of model weights (e.g., from 32-bit floats to 4-bit integers), which shrinks the model's size and can speed up inference."³ That is the deal, and for anyone running a model on a laptop it is a good one.

What the deal hides is plurality. One model is not one file.

For Llama-3.1-8B, llama.cpp's maintainers document roughly two dozen builds of the same weights, measured at 2.0042 bits per weight and 1.87 GiB at the smallest end, 4.8944 bits and 4.58 GiB in the middle, 8.5008 bits and 7.95 GiB above that, and 16.0005 bits and 14.96 GiB unquantized.³ Every one of them is honestly called Llama-3.1-8B. They are eight times apart in size.

So when two people compare notes about "the same model" and get different answers, the premise is already false. Your local model is a different model.

Flips: the thing an aggregate score cannot see

A benchmark score is a sum. If compression turns one correct answer wrong and one wrong answer right, the sum does not move at all. The NeurIPS team named that hidden churn — flips — and measured it across six quantization schemes and five chat models. Up to 13.6% of answers changed.²

Their sentence is worth reading exactly as they wrote it: "even when the overall accuracy is unchanged, a large number of correct answers change to incorrect and vice versa in proportion (we call these flips), between the baseline and compressed model."² The parenthesis is the coinage. Dutta, Krishnan, Kwatra and Ramjee believe they are, in their own careful phrasing, "the first to identify this phenomenon of flips caused due to model compression."²

The 13.6% is a maximum, not a law. It comes from one cell of the paper's Appendix A.1: Llama2-7b chat, SmoothQuant W8A8, MMLU at five shots — accuracy down 2.36 points against a 47.21 baseline, and 13.62% of answers flipped.² Quote it that way or not at all.

The same sentence carries its own counter-example, and the counter-example matters as much as the headline: "except for GPTQ W8A16 (8-bit weight, 16-bit activation Frantar et al. (2023)) that preserves accuracy with negligible flips, all other quantization schemes exhibit large number of flips ( up to 13.6%), indicating significant divergence from the baseline model."²

One scheme in the set stayed put. Compression is not uniformly destructive; it is unevenly so, and the score does not tell you which kind you downloaded.

The paper's abstract states the consequence plainly: "the behavior of compressed models as visible to end-users is often significantly different from the baseline model, even when accuracy is similar."²

Judge a build by distance, not by score

Accuracy answers "how capable is this model." The question you actually have is "how close is this file to the one I trust." Those are different measurements, and the paper argues the second is the right one: compressed models should be "better judged by distance metrics with respect to baseline."²

The full sentence gives the reasoning: "Since the goal of compression schemes is to create models that mimic the baseline models as closely as possible, we argue that compressed models are better judged by distance metrics with respect to baseline."² A compressed build is not trying to be smart. It is trying to be the same.

Distance is measurable, and the measurements agree with each other. KL-Divergence and the flips rate track closely — a Spearman correlation of 0.981 on MMLU.² Two independent ways of asking "how far did it move" return the same ranking.

Three findings from the same paper tell you where the movement concentrates.

More compression moves the model further. On MMLU, the 4-bit bitsandbytes build showed on average "2.4× more flips than" its 8-bit sibling.²

Harder questions move more than easy ones. "MMLU which is a relatively hard task has 8-16% flips for Bitsandbytes W4A4 whereas for the same technique, PIQA, an easier task, has 3-6% flips."² Asking a model to reconcile two of your meeting notes from different months is not the easy end of that range.

And the movement is not confined to multiple choice. On MT-Bench, which scores free-form generated text, "Most quantization methods degrade the MT-Bench score for the larger models, by 5% for Llama2-70b chat and 1.5% for Yi-34b chat" — and the second conversational turn suffers more, "with up to 10% loss for Llama2-70b chat and 5% for Yi-34b chat."²

Free-form answers are what you get from a model reading your notes. That is the closest bridge the literature offers from benchmark to vault, and it points the same direction.

None of this is one team's idiosyncrasy. Ajay Jaiswal and co-authors, presenting at ICLR 2024, built an entire evaluation suite on the premise that perplexity, the metric the compression literature leans on, can "fail to capture subtle change in their true capabilities" — while also finding that "current quantization methods are more successful than pruning."⁴ Quantization is the better-behaved compression family. Its divergence is easy to miss precisely because it is well-mannered on the scoreboard.

A 2026 preprint by Plawan Kumar Rath and Rahul Maliakkal, accepted for IEEE Cloud Summit 2026 but not yet published in its final form, reproduces the shape on newer models: perplexity "increases by less than 0.5% at 8-bit and under 3% at 4-bit across all three models, yet 2.5-5.6% of items already develop new biases at 4-bit."⁵ Different metric, different failure, same structure — the aggregate holds still while individual items move.

What this argument does not say

This is not an argument against running models locally. Local AI is viable, that case is already made, and nothing measured here retracts a word of it. The finding concerns which artifact you ran, never where you ran it. Four other readings are also wrong.

It does not say quantization ruins models. GPTQ W8A16 preserved both accuracy and stability in the same experiment.² The compression that lets a capable model fit on a phone is the same mechanism that shifts its answers; both facts are true, and the cheapest honest advice is to prefer the least-compressed build your hardware tolerates rather than to avoid compression.

It does not say flips mean degradation. The metric is symmetric by construction — it counts answers that turn wrong and answers that turn right — and the paper's own limitations section says as much: "Flips is only a warning that the behaviour of a model and its compressed version is different."²

That section also concedes the case where none of this matters: "If the downstream task is very similar to the benchmark on which the quantized model is tested, then accuracy may be sufficient, and distance metrics are not needed."²

It is not the run-to-run variance problem either. That one is one build rolled twice; this is two builds diverging under greedy decoding, where the dice were never in play. Fixing a seed does nothing for it. Nor is it the embedding-side model lock, where swapping the embedding model invalidates a stored index — a quantized generation model never touches the index.

And the honest limit: no study here measured a personal note vault. The benchmarks are 2024-era tasks on Llama-2, Yi, Qwen2 and Llama-3 chat models. What transfers is the mechanism and its direction, not the number. Nobody has published a flips rate for "questions about my own notes," and inventing one would be the same error the paper is complaining about.

What to do tomorrow

Record the build, test on your own questions, and prefer less compression when the task is recall over your own material. Three habits, all of them cheap, none of them requiring you to trust a leaderboard that was never measuring the thing you care about.

1. Write the build identifier next to any output you keep. Not "Llama-3.1-8B" — the file name, the quantization scheme, the bits per weight, the file size, and the checksum if you have one. This is the upgrade to pinning the model version in a note: for a local model, the version string is not the identifier. The file is.
2. Keep ten questions you actually ask your own notes. Save the answers from the build you currently trust. Re-run them whenever you swap builds and read the diff. That is a personal flips test, and it measures distance from your baseline rather than capability against someone else's.
3. Ask them as prose, not as multiple choice. The free-form gap was larger than the multiple-choice gap in the paper's own results, and larger again on the second conversational turn.²
4. Prefer the largest build your machine tolerates for anything where you care about the exact answer. More compression, more divergence — 2.4× more flips at 4-bit than 8-bit on MMLU.²
5. Re-check after any re-download. A repository can republish the same model name against a differently quantized file. Same label, different artifact, silently.

The whole discipline fits in one line appended to a note: the file, the scheme, the date. Plain text, on your own device, still readable when the runner you used this month is gone.

Frequently asked questions

Is a quantized model as good as the full one?

On the aggregate benchmark score, effectively yes — the NeurIPS 2024 study found all six schemes in its headline comparison landed within 0–2% of the 16-bit version.² On individual answers, no. Up to 13.6% of answers changed between baseline and compressed builds in the same experiment, with one scheme, GPTQ W8A16, as the exception.²

Does 4-bit quantization hurt accuracy?

Barely, by the score. It shifts behavior more than 8-bit does: on MMLU, the 4-bit bitsandbytes build averaged "2.4× more flips than" the 8-bit build.² For free-form text, MT-Bench scores dropped about 5% for Llama2-70b chat and 1.5% for Yi-34b chat across most quantization methods.²

Why does my local model give different answers than the hosted one?

Because you are almost certainly not running the same artifact. One model can ship as roughly two dozen documented builds — for Llama-3.1-8B, from 2.0042 bits per weight and 1.87 GiB up to 16.0005 bits and 14.96 GiB.³ The name on the download page describes the family, not the file.

What is the difference between Q4_K_M and Q8_0?

Precision and size. For Llama-3.1-8B, llama.cpp measures Q4_K_M at 4.8944 bits per weight and 4.58 GiB, Q8_0 at 8.5008 bits and 7.95 GiB.³ The more compressed build sits further from the 16-bit original, and divergence rises with compression.²

How do I know which quantization I am running?

It is in the file name or tag of the model you downloaded, not in the model's name. Read it once, then write it into the note beside anything you keep — file name, scheme, size, date. That record costs one line and is the only way to answer "which build produced this" a year later.

Does quantization make the model dumber?

Not on the score, and the researchers are careful about the stronger claim. Their limitations section says a flip "is only a warning that the behaviour of a model and its compressed version is different."² Answers move in both directions; the metric counts corrections as well as regressions. Quantization makes the model different, which is a smaller and more useful claim than worse.

How do I test whether a quantized build is good enough for my notes?

Keep about ten questions you genuinely ask your own material, save the answers from the build you trust, then re-run them against each new build and compare. You are measuring distance from your baseline, which is what the paper argues is the right measurement: compressed models are "better judged by distance metrics with respect to baseline."²

The label on the file tells you which family the weights came from. It does not tell you how the model will answer, and the score was never designed to. A benchmark measures capability; you needed similarity. Write down which build you ran — because a year from now, that line will be the only thing standing between an answer you kept and an answer you cannot account for.

The build identifier, the ten questions and their saved answers are just Markdown on your own device — the kind of record that outlives whichever runner you installed this month, in mnmnote.com.