Ask a Second AI to Check the First, and It Often Makes the Same Mistake

When two AI models get the same question wrong, they often give the same wrong answer. Across more than 350 models, a 2025 Cornell study found that on one leaderboard, pairs that both erred agreed about 60% of the time — random guessing would agree one time in three.¹ So a second model agreeing is weak evidence. Check the note.

A smaller, informal test made a related pattern visible this week. On 22 October 2026 the comparison site ModelCompare reported: "We asked 13 AI models for their most controversial opinion of us, three times each. 32 of 38 answers were the same one."² That was a question about opinions, with no right answer. The Cornell team measured something narrower and more useful — what happens on questions that do have a right answer, when the models miss it.

The habit this bears on is common. An AI summarises a long note, or answers a question from it. You paste the answer into a different model to double-check. The second model agrees, and you relax.

Diffing two AI answers shows where models disagree. This post is about the other half: what it means when they don't.

The study, briefly: hundreds of models, the same wrong answers

Elliot Kim, Avi Garg, Kenny Peng and Nikhil Garg, mostly at Cornell, compared the answers of hundreds of language models on two public leaderboards. On Helm, when two models both got a question wrong, they chose the same wrong option about 60% of the time. Choosing among the wrong options at random would match one time in three.

The paper, presented at ICML 2025, opens on the assumption it set out to test: "Diversity in training data, architecture, and providers is assumed to mitigate homogeneity in LLMs. However, we lack empirical evidence on whether different LLMs differ meaningfully."¹ Its answer, for Helm, is that "pairs of models agree on average about 60% of the time when both models are incorrect."¹

The second leaderboard, on HuggingFace, has questions with between three and ten options, so chance agreement there is lower: 0.127. The measured rate was 0.423.¹

Dividing one by the other gives 1.8 times the chance rate on Helm and about 3.3 times on HuggingFace — our arithmetic on the paper's numbers. The authors describe both as "about double or higher than the baselines."¹

This was not a few unlucky pairs. In the paper's words, "almost all pairs (100% of pairs on HuggingFace; 97.5% on Helm) of models have a higher agreement rate than the respective baselines."¹

What they actually measured

The numbers above come from multiple-choice questions drawn from MMLU, a standard academic benchmark. None of them comes from summarising a note or checking a claim in prose. The finding is about how models fail on questions with a known answer key; applying it to your notes is an inference — a reasonable one, and labelled as one here.

The authors flag their own limits. A wrong option is not always a random wrong option: "in practice, some incorrect answers may be 'closer' to correct, and so preferred by more accurate models; some questions may also be harder than others."¹ Part of the shared error, in other words, may be shared difficulty — a hard question trips everyone.

They also recorded which pairs agreed most, not why. "Models with the same provider (company), with the same base architecture, or with similar sizes have more correlated errors," they write.¹ Those are correlates — the paper does not show that shared training data causes the overlap.

The only proposed cause comes from a different study, and it is framed as a guess.

Liwei Jiang and colleagues, whose Artificial Hivemind paper won a NeurIPS 2025 best-paper award, measured how alike models sound on open-ended prompts. The average pairwise similarity between responses from different models "ranges from 71% to 82%."³ That is similarity of wording, not agreement on errors. On causes, they are candid: "Although the exact causes remain unclear due to proprietary training details, possible explanations include shared data pipelines across regions or contamination from synthetic data."³

Better models make more of the same mistakes

The obvious escape — use a stronger model as the checker — runs the wrong way. Kim and colleagues found that pairs of more accurate models have more correlated errors, even across different companies and architectures. A separate ICML 2025 team found the same direction: as capability rises, model mistakes grow more alike.

The Cornell abstract is direct: "larger and more accurate models have highly correlated errors, even with distinct architectures and providers."¹ The body adds the consequence: "newer models that differ on the surface may be converging in their outputs."¹

Shashwat Goel and eight co-authors reached the same place with a different metric, one built on overlap in model mistakes. Their paper's title states the finding — Great Models Think Alike and this Undermines AI Oversight — and the abstract names the risk: "model mistakes are becoming more similar with increasing capabilities, pointing to risks from correlated failures."⁴

Upgrading the checker does not buy independence. It may buy the opposite.

Why agreement fails as a check

A checker that shares the first model's wrong answer will mark that answer correct. Kim's team tested this directly: they took the most accurate model in a family as a judge, graded other models against its answers, and found each judge inflated the scores of weaker models — because it accepted the wrong answers it shared.

The paper spells out the mechanism: "each judge systematically inflates the accuracy of models that are less accurate than itself, due to correlated errors (the judge marks incorrect answers as correct if both models agree on the incorrect answer)."¹

The reverse also held. A judge marked down models more accurate than itself, because it "cannot reward a model for answering correctly on a question it itself answers incorrectly."¹ A second model can talk you out of a right answer, too.

Goel's team found a related pull: "LLM-as-a-judge scores favor models similar to the judge."⁴ Judges are known to carry other biases too — the order of the options, their length, a taste for their own prose — and asking twice with the order swapped covers those. The failure here is different. Nothing about the prompt is unfair. The judge simply shares the mistake.

The authors' own remedy is the one that transfers to notes. When measuring error rates with an AI judge, "it is important to calibrate error metrics for each model-judge pair using ground-truth data."¹

For a summary of your notes, the ground truth already exists. It is the note.

Engineers measured this in 1986

Software engineers tested the same assumption forty years ago. N-version programming runs several independently written programs and trusts the majority. In 1986 John Knight and Nancy Leveson had 27 versions built from one specification and ran a million tests. The programs failed together substantially more often than independence predicts.

Their abstract names the axiom under test: "This method depends for its reliability improvement on the assumption that programs that have been developed independently will fail independently."⁵ And the result: "the programs were individually extremely reliable but that the number of tests in which more than one program failed was substantially more than expected."⁵

Independently built is not independently failing.

The US National Institute of Standards and Technology carried the point to generative AI in its 2024 risk profile. It warns that shared foundation models can act as "bottlenecks," or single points of failure, and that "overly homogenized outputs can themselves be incorrect, or they may lead to unreliable decision-making or amplify harmful biases."⁶

What a second model is still good for

Disagreement still carries information, and combining models can help. Correlation is high, not total: when two models both miss, they miss differently about 40% of the time on Helm. In a 2024 ICML study, model instances that debated reduced hallucinations; in a 2024 Science Advances study, a crowd of 12 models forecast about as accurately as 925 human forecasters.

Yilun Du and colleagues had several model instances propose answers and argue over rounds, and report that the approach "improves the factual validity of generated content, reducing fallacious answers and hallucinations that contemporary models are prone to."⁷

Philipp Schoenegger, Philip Tetlock and co-authors pooled 12 models on 31 forecasting questions: "We show that the LLM crowd outperforms a no-information benchmark and is statistically indistinguishable from the human crowd."⁸

So the strong version of the worry goes too far. Responding to the ModelCompare post, one Hacker News commenter concluded that "asking models from different companies to review or cross-check each other is probably useless."⁹

Not useless. Lopsided.

When a second model disagrees, it has found a place where at least one of them is wrong, and that is worth knowing. When it agrees, it has told you much less than it seems to — the two were likely to share the error in the first place.

The same holds for a panel of personas played by one model — Jiang's team found repetition within a single model as well as across them.³ An advisory board in a Markdown file is useful for raising angles, not for confirming facts.

How to apply this tomorrow

Use a second model to locate claims, never to certify them. Let it list what the first answer asserts and flag what it doubts. Treat any disagreement as a place to look. Treat agreement as weak evidence, not a verdict. Then settle each claim that matters against the note itself.

1. Ask for claims, not a verdict. Instead of asking whether the summary is correct, ask the second model to list each factual claim in it, one per line. A list is checkable. A yes is not.
2. Read disagreement as a flag. Where the two answers differ, diff them and look at the note for that line first.
3. Do not upgrade to certify. Pairs of stronger models share more errors, not fewer.
4. Check the line in the note. Have the AI quote the exact sentence and the note it came from, then search your files for it. If the sentence is not on disk, the claim was not grounded in your notes.
5. Mark what no note supports. A date, a figure or a name the model supplied from outside your notes is exactly the case this research describes. Two models agreeing on it adds little. Label it unverified until you find a source.

Phrase the check neutrally, too. A model asked to confirm tends to confirm — the subject of leading questions about your notes. And a check is only worth as much as its independence from the thing being checked, the lesson of agents that optimise the check instead of the work.

The caveats

The research is strong on one point and silent on others. It shows that model errors are correlated on multiple-choice benchmarks. It does not measure summaries or notes, it does not establish why models converge, and its models predate mid-2025. Newer models could differ, though both 2025 papers found errors converging as capability rose.

• Multiple choice is not your notes. The 60% figure comes from MMLU questions on two leaderboards. The rule for notes rests on the judge mechanism, which does not depend on the question format, but no one has measured it on personal notes.
• Correlates, not causes. Provider, architecture, size and accuracy track shared errors. Shared training data is a hypothesis — offered by the Hivemind authors as a possible explanation, not a finding.
• Similar is not wrong. The 71–82% Hivemind figure measures how alike open-ended answers sound. It is not a second error rate.
• The hook is informal. ModelCompare's 32 of 38 answers were opinions, from one prompt, with no answer key.
• Correlation is not identity. Debate and aggregation helped in measured settings. Disagreement remains a useful signal.

Frequently Asked Questions

Do different AI models make the same mistakes?

Often, yes. A 2025 Cornell study of over 350 models found that when two models both got a multiple-choice question wrong, they picked the same wrong answer about 60% of the time on one leaderboard, against one in three by chance. Nearly every pair of models agreed on errors more than chance predicts.

Should I use two AI models to verify an answer?

Use the second model to find claims and flag disagreements, not to confirm the answer. Disagreement points to a likely error. Agreement is weak evidence, because models tend to share mistakes. For anything that matters, check the claim against the note or document it is supposed to come from.

If two AI models agree, is the answer correct?

Not necessarily. Agreement shows the models reached the same answer, and models share errors far more often than chance. In one test, a judge model graded other models against its own answers and marked shared wrong answers as correct. Agreement makes an answer more consistent, not more verified.

Is using an AI as a judge reliable?

Only partly. Cornell researchers found AI judges inflate the scores of weaker models because they accept wrong answers they share, and a separate ICML 2025 study found judges favour models similar to themselves. The Cornell authors recommend calibrating any AI judge against ground-truth data — for notes, the note itself.

Does a smarter model make a better checker?

Not for independence. Kim and colleagues found that more accurate models have more correlated errors, even across companies and architectures, and Goel and colleagues found mistakes growing more similar as capability rises. A stronger checker may be right more often, but when it is wrong, its mistakes are more likely to match another strong model's.

Is cross-checking AI models useless, then?

No. Correlation is high, not total — on one leaderboard, models that both erred still picked different wrong answers about 40% of the time. Multiagent debate reduced hallucinations in a 2024 ICML study, and a crowd of 12 models forecast about as well as 925 humans. The weak part is treating agreement as confirmation.

How do I fact-check an AI summary of my notes?

Ask the AI to list each claim and quote the exact sentence and note it relied on. Then search your files for that sentence. If it is there, read the surrounding lines; if it is not, the claim was not grounded in your notes. A second model can help find claims, but only the note confirms them.

Two models that agree have told you they think alike; only the note can tell you they are right.

mnmnote.com keeps your notes as plain Markdown files on your own device, where any sentence an AI attributes to them can be searched for in seconds.