If You Don't Write Your Notes in English, AI Charges You More

If your notes are not in English, an AI charges you more to read them — not because the model judges your language, but because your file is cut into more pieces before the model runs. That premium has roughly halved since 2023. It has not reached parity for a single language.

The unit of that charge is the token, and tokens are not letters. Take the Shan word မႂ်း, one of the Shan words for "you." In 2023, ChatGPT and GPT-4's tokenizer cut it into nine tokens. The paper that measured it prints the mechanism plainly: the word "is constructed from one consonant and three diacritics," the diacritics encode separately, "there are four Unicode codepoints for this Shan character, resulting in 9 tokens." Then the line that does the damage: "The English 'you' has three characters but a single token."¹ Nine against one, for the same word, in the same sentence, before a model has read a thing. Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr and Adel Bibi measured this across 200 languages and published it at NeurIPS 2023.² Their headline: the same text translated into different languages can differ in tokenization length "up to 15 times."²

That 15× is real. It is also three years old, and measured on a tokenizer most people have stopped using. Both halves of that sentence matter.

The belief: the model has an opinion about your language

Ask an AI to summarise a page of Hindi notes and it feels slower, pricier, and blunter than the same page in English. The natural conclusion is that the model was trained mostly on English and simply understands your language less well. That conclusion is reasonable, widely held, and lands exactly one layer too late.

The model's training mix is a real thing, and it does shape output quality. But it cannot explain a bill that is already larger before the model is consulted. Something happens to your file first — upstream of the weights, upstream of the prompt, upstream of anything anyone would call intelligence. It happens in the boring part of the pipeline nobody writes essays about.

The vendors say so themselves, in a clause most people skim. Anthropic's pricing FAQ offers the familiar rule of thumb — "1 token is approximately 4 characters or 0.75 words in English" — and then adds, without elaborating: "The exact count varies by language and content type."³ That trailing sentence is this post's entire subject. Token cost is the new page count for your notes takes the rule of thumb at face value and prices the paste. This post is about the clause at the end of it.

The mechanism: it happens to your file, before the model runs

Petrov and colleagues locate it. Their abstract states that "we show how disparity in the treatment of different languages arises at the tokenization stage, well before a model is even invoked."² Tokenization is where your text is chopped into the units a model bills for. The chopping happens to the file. The model is not present.

Subword tokenizers are fitted to whatever text their makers had most of. Sequences that appeared often in that mix get one token; sequences that did not get broken into fragments. This is not a judgment. It is a compression artifact — a lookup table that learned English well and your language less well, and now charges by the piece.

Which is why the effect is so stubbornly mechanical. The premium does not care what you wrote, only what script you wrote it in. Petrov et al. define the measurement as a plain ratio: take a sentence in language A and its translation into language B, and "the ratio |t(sA)|/|t(sB)| is the premium for A relative to B."⁴

Parity, in their words, "occurs when a tokenizer exhibits similar tokenized lengths for the same sentence in different languages."⁴ No tokenizer they tested achieved it.

What the premium actually costs you

The authors name three consequences. Tokenization disparity, they write, "induces unfair treatment for some language communities in regard to the cost of accessing commercial language services, the processing time and latency, as well as the amount of content that can be provided as context to the models."² Cost, latency, capacity.

The bill

For anyone paying per token, the premium is the bill. Petrov et al. put a floor under it on 2023's tokenizer: these discrepancies, they write, "lead to users of some languages paying at least 2.5 times more for the same task as users of English."⁵

Orevaoghene Ahia and co-authors reached the same conclusion independently at EMNLP 2023, measuring OpenAI's API across 22 typologically diverse languages and finding "a large variance in the average number of tokens required to convey the same information with some languages requiring 5 times as many tokens than others."⁶ Note the phrasing: than others, not than English.

The wait

Tokens are also the unit of work. "Some languages can require twice the time to process the same content as English," Petrov et al. reported in 2023.⁵ The model is not thinking harder about your language. There is simply more of it to walk through.

The room

This is the one that reaches your vault. A context window is a fixed number of tokens, identical for everyone. What changes is how much of your material fits inside it. Petrov et al., writing in 2023: "Users of languages that are more token-efficient can use these systems to process or generate texts that may be more than an order of magnitude longer than users of other languages."⁵

Two people paste the same window full; one gets a chapter, the other gets a paragraph. The context window is working memory is the ceiling — the premium decides how much of your thinking fits under it.

Ahia et al. go one step further, and it is the ethical centre of the research rather than a lever for anybody's product. They found that "speakers of a large number of the supported languages are overcharged while obtaining poorer results. These speakers tend to also come from regions where the APIs are less affordable to begin with."⁷

They coined a name for it: "We term this 'double unfairness' as people from less economically developed countries are overcharged at a fixed rate per-token due to excessive tokenization, but often derive less utility from the model."⁸ That finding is about speakers in less economically developed countries. It is not about note apps, and it should not be borrowed as one.

The 2023 number is not the 2026 number

Most retellings of this research stop too early. "Up to 15 times" was measured on cl100k_base — the tokenizer behind ChatGPT and GPT-4 in 2023.² Today's OpenAI models use o200k_base. Re-running Petrov's method on the current tokenizer gives a different picture: the gap has roughly halved, and no language has reached parity.

The method is theirs, unchanged: the mean per-sentence ratio against English over all 2,009 parallel sentences of FLORES-200 (dev + devtest) — "the same 2000 sentences taken from Wikipedia and human-translated to 200 different languages."^{5 9}

The reason to trust the second column is the first. Run on cl100k_base, this method reproduces Petrov's published Table 1 across all 14 overlapping languages, with a maximum deviation of 0.28. Reproduce the old number, then measure the new one.

Premium vs English, mean of per-sentence ratios, 2,009 parallel FLORES-200 sentences, measured 2026-07-17 with tiktoken 0.13.0.^{9 10}

Read that table honestly and it says two things at once, and you need both.

A vendor shipped a real fix. Across the two tokenizer generations — 2023's cl100k_base to 2026's o200k_base — Hindi fell from 4.83 to 1.59. Arabic from 3.06 to 1.39. Burmese from 11.83 to 3.21. The Shan word that cost nine tokens in 2023 costs five today. The vocabulary roughly doubled between the two, from 100,277 tokens to 200,019, and the disparity narrowed by about half.¹⁰ The villain in this story is subword encoding, and encoding gets better.

And nothing reached parity. Zero of the nineteen languages measured here landed at 1.00. English is 1.00 by construction; every other language pays. Portuguese — the best case in the paper and still the best case now — went from about 50% more tokens in 2023¹ to 23% more in 2026.

That is the floor of this argument, not the ceiling. The interesting number is not the worst case; it is that even the best case still pays. Petrov et al. saw the durability coming: "These disparities persist even for tokenizers that are intentionally trained for multilingual support."²

Two honest caveats. Nineteen languages is not two hundred, so 8.10 is the worst case in this sample — Shan was Petrov's worst on the old tokenizer, but the global worst on o200k_base may be a language not measured here. And o200k_base is OpenAI's; this table says nothing about Anthropic's or Google's tokenizers.

Why the obvious fixes don't work

Three escape hatches suggest themselves, and the research closes all three. You cannot encode your way out, you cannot buy your way out, and you cannot wait for a multilingual tokenizer to save you — because the multilingual ones were measured too.

"Use a byte-level model." Petrov et al. checked: "Character-level and byte-level models also exhibit over 4 times the difference in the encoding length for some language pairs."² Smaller, still not one.

"Use a bigger context window." The premium is a ratio, and a ratio does not care how big the window is. Everyone gets the same window; your notes are simply longer inside it.

"Use a multilingual tokenizer." That is the one Petrov et al. explicitly tested and rejected as a cure, in the sentence quoted above.² Deliberate multilingual training narrows the gap. It has not closed it.

Which leaves the one thing the papers agree on, and it is not addressed to you. Their recommendation is that "we should train future language models using multilingually fair subword tokenizers."² That is a message to the people who build tokenizers. You are not one of them. You have a vault and a bill.

What to do: measure your own coefficient

The useful move is not to change how you write. It is to learn your own number, because the published averages are about FLORES sentences and you do not write FLORES sentences. Your ratio depends on your language, your script, your jargon, and your formatting. It is a property of your files, and it is measurable in about a minute.

1. Measure it. Take one real note. Take its English translation. Count both with the same tokenizer, and divide.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")        # GPT-4o / GPT-5
ratio = len(enc.encode(your_note)) / len(enc.encode(english_translation))

That quotient is your coefficient — the thing every token estimate you read online silently assumes is 1.00.

2. Do not rewrite your thinking in English. Nothing in either paper supports it. The cost of writing in a second language is real and is not denominated in tokens; a worse note in cheaper units is a bad trade. The lever is what leaves your device, not what language you think in.
3. Send less, not different. The premium multiplies whatever you send, so trimming is worth more to you than to an English speaker. Shape your prompt before you send it is the lever; your coefficient is the multiplier on it.
4. Know which tokenizer you are on. Vendors differ, and that is a separate axis with its own arithmetic — token cost is the new page count for your notes covers which vendor you send to. This post covers the coefficient that multiplies every vendor's number at once.
5. Re-measure when the tokenizer changes. The table above has a date on it for a reason. o200k_base replaced cl100k_base and halved the gap; the next encoding will move it again.

None of this is fixable from your side. All of it is knowable from your side, and knowable is worth more than it sounds. Plain text is safe from mojibake at the byte layer — your file is fine, your bytes are intact. One layer up, that same file is quietly expensive, and the only way to find out by how much is to count your own.

Frequently asked questions

Does it cost more to use ChatGPT in another language? Yes, in tokens. Petrov et al. found users of some languages pay "at least 2.5 times more for the same task as users of English" on 2023's tokenizer.⁵ On OpenAI's current o200k_base, the measured premiums range from 1.23× (Portuguese) to 8.10× (Shan) across the nineteen languages measured here. Your own ratio is measurable in a minute.

Why does my non-English text use more tokens? Because tokenization happens to your file before any model runs.² Subword tokenizers are fitted to whatever text their makers had most of; frequent sequences get one token, rare scripts get fragmented. One Shan word costs nine tokens on 2023's tokenizer because its four Unicode codepoints encode separately.¹

Has this gotten better? Measurably, yes. Between cl100k_base (2023) and o200k_base (2026), the premium roughly halved: Hindi 4.83 → 1.59, Arabic 3.06 → 1.39, Shan 15.33 → 8.10.¹⁰ A vendor shipped a better tokenizer and the gap narrowed. No language reached parity — the best case, Portuguese, still pays 23% more.

Can I just use a byte-level model to avoid it? No. Petrov et al. measured them: character-level and byte-level models "also exhibit over 4 times the difference in the encoding length for some language pairs."² Byte-level encoding compresses the disparity. It does not remove it, because the disparity is in the scripts, not only in the vocabulary.

Should I write my notes in English to save money? No. Neither paper supports it, and it misprices the trade. The cost of thinking in a second language is real and is not measured in tokens — you would be paying in the quality of the note to save on the encoding of it. Send less; do not think differently.

Will a bigger context window fix it? No. The premium is a ratio, not a fixed overhead. The window is the same size for everyone; your notes are simply longer inside it. That is precisely Petrov et al.'s third consequence, measured on 2023's tokenizer: token-efficient languages fit "more than an order of magnitude longer" texts in the same context.⁵ On o200k_base the widest gap in this sample is 8.10×.

Are LLMs more expensive for non-English speakers? Ahia et al. found speakers of many supported languages are "overcharged while obtaining poorer results," and that they "tend to also come from regions where the APIs are less affordable to begin with."⁷ That was measured on multilingual NLP benchmarks against OpenAI's API — a finding about API pricing and benchmark utility, not about retrieval quality in anyone's private notes.

The asymmetry

The fix the researchers propose is not available to you. Petrov et al. close by arguing that "we should train future language models using multilingually fair subword tokenizers"² — a sentence addressed to model builders. It is the right recommendation. It is also somebody else's to implement, on a timeline nobody has published.

The last three years suggest it arrives the way it arrived this time: as a quiet vocabulary change in a release note, not as a policy. What you get in the meantime is a coefficient. It came down by half without your involvement, it will move again without your permission, and the only place it is ever true about your writing is on your own files.

A tokenizer is not an opinion about your language — it is an artifact of what its makers had most of, and the only part of the arrangement you control is what you hand it.

Notes kept as plain Markdown on your own device can be counted, diffed, and withheld one at a time — which is what makes the coefficient measurable at all, and it is how MNMNOTE keeps them.