Ask an AI Which Version Is Better, Then Swap the Order

Ask an AI which version of your note is better, and part of the answer depends on which version you pasted first. In the 2023 MT-Bench study, GPT-4 gave the same verdict on only 65.0% of answer pairs once their order was swapped.¹ So ask twice, with the order swapped. If the verdict flips, call it a tie.

That rule is not ours. It is the method the MT-Bench authors — a team from UC Berkeley and partner institutions, the group behind Chatbot Arena — chose for their own experiments: "A conservative approach is to call a judge twice by swapping the order of two answers and only declare a win when an answer is preferred in both orders. If the results are inconsistent after swapping, we can call it a tie."¹

The everyday version of the problem is familiar. You wrote a note; you asked an AI to tighten it; now you have two versions and you ask the same AI which version is better. Diffing the two shows you what changed — "not which version is right; that judgment stays yours." This post is about what happens when you hand that judgment to the AI anyway.

Three things lean on the verdict besides quality: the order of the versions, their length, and which one the model recognises as its own. The fix is a short routine. It extends the flip-order test — written for answers drawn from your notes — to verdicts between two candidates.

Order: the first version often starts ahead

When an AI compares two versions, the slot each one sits in can decide the verdict. MT-Bench asked three 2023 judges to compare answer pairs in both orders. Claude-v1 gave a consistent verdict on 23.8% of pairs and favoured the first answer 75.0% of the time. GPT-3.5 managed 46.2%; GPT-4, 65.0%.¹

The authors put it bluntly: "Only GPT-4 outputs consistent results in more than 60% of cases."¹ Consistency here has a strict meaning — "the percentage of cases where a judge gives consistent results when swapping the order of two assistants."¹ Everything outside that share was a verdict that followed the slot rather than the answer, or a malformed reply.

Prose is where order decided most.

The paper's appendix breaks the results down by question type, without naming which judge produced that table.² On writing questions, the judge stayed consistent on 42.0% of pairs and favoured the first answer on 46.0%. On humanities questions it held 36.0%. On math and coding it held 86.0%. The authors' summary: "position bias is more noticeable on open questions like writing and stem/humanity knowledge questions."² A note is an open question — there is no answer key to check against.

The second pattern matters even more for notes. Position bias, the appendix found, "is more noticeable for models with close performance and can almost disappear when the performance of the two models differs a lot."² When GPT-4 judged GPT-3.5 against Claude-v1 — two close models — it was consistent on 67.5% of pairs. Against the much weaker LLaMA-13B, 98.8%.²

A note and its light rewrite are the close case. They share most of their sentences — the same condition the authors flagged in their own test pairs: "Note that this test is challenging because the answers are very similar and occasionally indistinguishable even to humans."¹

The closest measured task to a note is a summary. In Panickssery, Bowman and Feng's NeurIPS 2024 study of news-article summaries, "GPT-4, GPT-3.5, and Llama reverse their pairwise preferences when the ordering of options is reversed at rates of 25%, 58%, and 89% respectively, averaged across tasks and datasets."³

None of this makes the AI uniquely unreliable. The MT-Bench authors note that "This bias is not unique to our context and has been seen in human decision-making."¹ A human panel would get the same routine. The AI just makes it cheap to run.

Every row is a measurement on a named model of its time — not a rate for the model you use today.

Length: the longer version gets credit for being longer

AI judges tend to prefer the longer of two versions, even when length adds nothing. MT-Bench defines verbosity bias as a judge favouring "longer, verbose responses, even if they are not as clear, high-quality, or accurate as shorter alternatives."¹ The vendor that makes GPT-4 says the same thing in its own evaluation guide.

The sharpest test was an attack, and it should be read as one. MT-Bench took 23 list-style answers and padded each with rephrased copies of its own items — more words, nothing new.¹ An attack "is successful if an LLM judge thinks the new response is better than the old response."¹ The padded version won 91.3% of the time with Claude-v1 and GPT-3.5. GPT-4 fell for it 8.7% of the time.¹

That is a worst case, not a general rate. The general pattern comes from other labs.

Saito and colleagues at the University of Tsukuba, RIKEN and LY Corporation define verbosity bias as "a bias where LLMs sometimes prefer more verbose answers even if they have similar qualities." They found that "in our problem setting, GPT-4 prefers longer answers more than humans."⁵

Stanford's AlpacaEval team showed the flip side: correcting for length "increases the Spearman correlation with LMSYS Chatbot Arena from 0.94 to 0.98."⁶ Take length out, and the automated judge moves closer to what people prefer.

OpenAI's own guide names both problems in one line — "Position bias (response order), verbosity bias (preferring longer responses)" — and advises: "Control for response lengths as LLMs bias towards longer responses in general."⁷

For a note, the trap is quiet. If the rewrite added a sentence of context, a summary line, or a friendlier opening, it is longer. "B is more thorough" can be a real judgement. It can also be the length talking.

"Mine": the judge may favour text it recognises

There is evidence that AI judges favour text they recognise as their own, though it is the weakest of the three effects. MT-Bench's own test was inconclusive. A 2024 NeurIPS study found a link between a model recognising its writing and preferring it — and on one dataset, a label claiming authorship was enough to shift the verdict.

Panickssery, Bowman and Feng define the effect as "self-preference, where an LLM evaluator scores its own outputs higher than others' while human annotators consider them of equal quality."³ Fine-tuning models to recognise their own writing, they report, revealed "a linear correlation between self-recognition capability and the strength of self-preference bias."³

Then they swapped the labels. On the XSUM summaries, judges "show a preference for the summary labeled as being their own generation" — even when the label was false. The same paragraph is careful: "In the CNN dataset or the Llama evaluator, no clear difference is shown from label reversal."³

MT-Bench, a year earlier, could not settle the question at all. GPT-4 favoured itself with a 10% higher win rate and Claude-v1 with 25% — but both also favoured other models, and GPT-3.5 did not favour itself. "Due to limited data and small differences, our study cannot determine whether the models exhibit a self-enhancement bias."¹

So treat it as a risk, not a law. The fix costs nothing anyway.

When you ask "is my version or your rewrite better?", you hand the judge two labels — one of them pointing at itself. Remove them. Call the versions A and B. Better still, ask in a fresh chat that never saw the rewrite being written.

The single habit: ask twice, swapped, and call a flip a tie

Ask the same question twice, with the two versions in opposite order, and accept a winner only if it wins both times. If the verdict changes when the order changes, record a tie. This is MT-Bench's own conservative method, and it is our flip-order test applied to a verdict instead of an answer.

It works without knowing your model's bias rate. A judge that leans toward the first slot leans the same way in both asks — so its lean cancels, and only a preference that survives the swap gets through. A position preference cannot pass the test twice in opposite directions.

A tie still tells you something. Recall the closeness finding: order decides most when the two versions are near each other. A flip says the versions are close enough that the slot outweighed the difference. At that point the choice belongs to your purpose, not to the model.

The cost is one extra ask. MT-Bench's authors went further: "In the following experiments, we use the conservative one."¹ Engineers who route yes/no decisions through a model meet the same advice in What is SemIf? — score twice with the option order reversed.

The five-minute version

The quick routine takes two asks and a one-line purpose. Write what the note is for, label the versions A and B with no hint of who wrote which, and ask for A, B or tie. Then ask again in a fresh chat with the versions reversed. Same winner twice is a win; anything else is a tie.

1. Write the purpose first. One sentence: "I need this note to remind me, in six months, why we chose the second supplier." A judge with no purpose falls back on generic taste — which is where length and polish win.
2. Strip the labels. No "mine", "original", "your rewrite". Call them Version A and Version B.
3. Ask once. Use the prompt below, with your note first.
4. Open a new chat. A model that can see its first verdict is no longer answering blind — the second ask has to start clean.
5. Ask again, swapped. Same prompt, same purpose, the rewrite now in the first slot.
6. Read the two verdicts. Same version both times: a win. Different versions, or a tie in either ask: a tie.
7. On a tie, decide by the purpose yourself — or keep the sentences you prefer from each.

The prompt, ready to paste:

Purpose: I need this note to [what the note must do, for whom, and when].

Below are two versions of the same note, labelled A and B.
Judge only which version better serves the purpose above.
Ignore length unless it helps or hurts the purpose.

Answer with exactly one of: A, B, or TIE.
Then give one sentence naming the specific difference that decided it.

--- Version A ---
[paste the first version]

--- Version B ---
[paste the second version]

For the second ask, swap what goes under A and B. Leave everything else identical — a changed word in the prompt is a second variable.

The one-sentence reason is a check, not decoration. If both asks name the same concrete difference, the verdict has something to stand on. If the reasons disagree — or name "more detailed" when you can see it is just longer — the verdict is weaker than its letter.

The thirty-minute version: for notes that matter

For a note that will steer a decision, ask about one property at a time instead of "which is better?". Pick the two or three things the note must get right — accuracy against the source, the decision it records, clarity for a stranger — and run the swapped pair for each. Then read both versions yourself.

OpenAI's guide makes the case for narrow questions: "LLMs are better at discriminating between options. Therefore, evaluations should focus on tasks like pairwise comparisons, classification, or scoring against specific criteria instead of open-ended generation."⁷ "Which is better?" is open-ended. "Which version states the deadline correctly?" is a specific criterion.

Equalise length where you can. If the rewrite is a third longer because it added a summary, compare the bodies with the summary removed — then decide separately whether you want the summary. That isolates the change you are judging.

Run the swap per property. Two asks per question, fresh chat each time. Three properties is six asks — still under ten minutes of pasting.

Diff before you judge. Diffing the two versions tells you exactly which sentences changed. Most "which is better?" questions shrink to two or three changed lines, and you can judge those yourself faster than any model.

Worked examples help some judges — with a catch. MT-Bench found that giving GPT-4 a few judged examples raised its swap consistency from 65.0% to 77.5%. For GPT-3.5 it was a trade, not a fix: the appendix says few-shot judging "moves the position bias of GPT-3.5 from the first position to the second position."² The main text adds that "high consistency may not imply high accuracy."¹ Keep the swap even with examples.

Anything that matters, read both. The rule from our plain-text eval post holds here: "do not hand the diff to an AI and call it judged." The routine makes the AI a better second reader. It does not make it the first.

Common mistakes

Most bad verdicts come from the setup, not the model. Asking in the chat where the rewrite was written, labelling one version as yours, asking "which is better?" without a purpose, and re-asking until you like the answer all let something other than quality decide. Each one has a cheap fix.

• Judging in the same chat that wrote the rewrite. The model can see which version it produced. Start fresh.
• Labelling the versions. "Mine" and "yours" are labels; Panickssery's XSUM experiment found a false label alone shifted preference.³ Use A and B.
• Treating "more thorough" as "better". Check whether the winning version is simply longer. Padding beat two of three 2023 judges in the 91.3% attack.¹
• Asking with no purpose. Without a target the judge picks on generic polish. One sentence of purpose changes the question it answers.
• Re-asking until it agrees with you. The routine is two asks and a rule fixed in advance. A third, fourth and fifth ask, stopped when you like the answer, is a vote you ran yourself.

What the numbers can and cannot tell you

The striking numbers in this post come from 2023 and 2024 models. Newer judges look steadier, but no study shows the bias has gone. A 2024 study of 15 judges found the best still kept their verdict on only about 76 to 82% of pairs. A small 2026 test saw one flip in twenty. Neither tells you your model's rate.

Shi and colleagues at Dartmouth ran a systematic study of the effect: "15 LLM judges across MTBench and DevBench with 22 tasks and approximately 40 solution-generating models, result in over 150,000 evaluation instances."⁴ Their conclusion was that position bias "is not due to random chance and varies significantly across judges and tasks," and is "strongly affected by the quality gap between solutions."⁴

On MTBench pairs, the best-scoring judges they tested — Claude-3.5-Sonnet and GPT-4 — gave the same verdict after a swap 82% of the time on average; GPT-4o and o1-mini, 76%.⁴ Better than 2023. Still roughly one pair in five.

In October 2026, a small self-published study tested one current judge on 20 prompts. It reported that "reversing candidate presentation order preserved the underlying preferred response in 19 of 20 comparisons," and declined to generalise in either direction: "The evidence also does not establish the opposite claim that GPT-5.6 Sol is generally position-robust."⁸

That is the honest state of the evidence. The measured rate has narrowed across model generations. It has not been shown to be zero.

You cannot know your own judge's rate on your own notes without asking twice. Which is the routine.

The other limits, plainly. Several MT-Bench results rest on small samples — the padding attack used 23 answers. The self-preference evidence comes mainly from one summary dataset. And none of these studies used personal notes. Notes are short, informal and close in wording to their rewrites, which is the condition that made order matter most.

Frequently Asked Questions

These are the questions people type when an AI's verdict between two drafts feels off. Each answer is short enough to act on and points back to the study behind it. The numbers describe the models that were tested, mostly from 2023 and 2024, so read them as evidence that the effect exists — not as your model's rate.

Is ChatGPT biased when comparing two texts?

Measured AI judges have been. In MT-Bench's 2023 tests, GPT-4 gave the same verdict on 65.0% of swapped pairs and GPT-3.5 on 46.2%. A 2024 study of 15 judges found the best still kept their verdict on only about 76–82% of pairs. Ask twice with the order swapped and treat a flip as a tie.

Does the AI prefer the first option?

Often, though not always. In MT-Bench, Claude-v1 favoured the first answer 75.0% of the time, GPT-3.5 50.0% and GPT-4 30.0%. A 2024 Dartmouth study found the effect varies significantly across judges and tasks. The MT-Bench authors note that humans show position bias too, so the fix is the same: swap and re-ask.

Does the AI prefer its own writing?

Possibly. A NeurIPS 2024 study found that models which better recognise their own text also prefer it more, and that labelling a summary as the model's own shifted its verdict on one dataset — but not another. MT-Bench could not confirm the effect. Remove "mine" and "yours" labels and ask in a fresh chat.

Does the AI prefer longer answers?

Yes, measurably. A 2023 study found GPT-4 "prefers longer answers more than humans" in its setting, and OpenAI's evaluation guide advises controlling for length. In MT-Bench's padding attack, rephrased duplicate list items fooled Claude-v1 and GPT-3.5 91.3% of the time. Compare equal-length passages or ask about one specific property.

How do I get an unbiased AI comparison of two drafts?

Write one sentence of purpose, label the drafts A and B, and ask which better serves that purpose. Then open a new chat and ask again with the order swapped. Accept a winner only if it wins both times. For important notes, ask about one property at a time and read both drafts yourself.

If the answer flips when I swap the order, which one is better?

Neither, as far as the AI can tell — call it a tie. MT-Bench uses exactly that rule. A flip usually means the two versions are close, which is when order matters most. At that point decide by your purpose: keep the version that does what the note is for, or merge the best sentences from each.

A verdict that changes when the pages change places was never about the pages.

mnmnote.com keeps both versions of a note as plain Markdown files on your own device, ready to paste in either order.