Python 3.15 Reads Your Notes as UTF-8 Now — Check the Scripts That Wrote Them

Python 3.15, released on October 9, 2026, reads and writes text as UTF-8 whenever a script omits encoding=. On Windows that mostly fixes things: the old 'charmap' codec can't decode error on UTF-8 notes goes away. What it breaks is the other half — notes your older scripts already wrote in the Windows code page.

The change is small to state and awkward to check. The script did not change, and neither did the files. Only the interpreter's guess about bytes moved — so a tag fixer that ran cleanly for years can stop on its first accented note.

Linux, and macOS with a UTF-8 locale, already read text as UTF-8 — nothing changes there for file reads. A note that holds only ASCII characters is stored as the same bytes in UTF-8 and in cp1252, so it is unaffected. The risk sits in one place: non-ASCII characters written on Windows by a script that never named an encoding.

This post walks the mechanism — the old default, the error 3.15 removes, the error it adds, and the audit PEP 686 prescribes. It ends with an eleven-line scanner for notes already on disk. Python 3.15 reads your notes as UTF-8 now; the job is to check the scripts that wrote them.

What changed in Python 3.15, and on which machines

Before 3.15, open() without an encoding= argument used the locale encoding. On Windows that is the ANSI code page — cp1252 on a Western European install; elsewhere it is usually UTF-8. Python 3.15 turns on UTF-8 mode by default, so every platform now reads and writes UTF-8 unless a script says otherwise.

The release notes put it in one line: "Python now uses UTF-8 as the default encoding, independent of the system’s environment."¹ The built-in open() documentation spells out the rule a script now follows: "In text mode, if encoding is not specified, UTF-8 is used by default; if Python UTF-8 Mode is disabled, locale.getencoding() is called to get the current locale encoding."²

What locale.getencoding() returns is the crux — and the Python docs answer it for the platform that matters: "On Windows, return the ANSI code page."³ Microsoft's code-page table lists 1252 as "ANSI Latin 1; Western European (Windows)", 1251 as Cyrillic, and 932 as Japanese Shift-JIS.⁴ So "the old default" was never one encoding. It was whichever code page the writing machine was set to — and the bytes on disk still follow it.

PEP 686, which made the change, names the files most likely to suffer. Its author, Inada Naoki, writes that developers "omit to specify encoding="utf-8" when they read text files encoded in UTF-8 (e.g. JSON, TOML, Markdown, and Python source files). Inconsistent default encoding causes many bugs."⁵

Markdown is on that list by name.

The split surprised people on release day. In the Hacker News thread for the 3.15 release, one reader wrote, "I really thought UTF-8 was already the default."⁶ The reply was the whole story in eight words: "On Linux it was, but not on Windows."⁷

The error Python 3.15 removes: 'charmap' can't decode byte 0x9d

On Windows, the error people have met for years runs the other way. A pre-3.15 script read a UTF-8 note with cp1252 and either garbled it or stopped on 'charmap' codec can't decode byte 0x9d. Stack Overflow's canonical question on that message has a score of 1155. Under 3.15, the same script reads the same note correctly.

The byte in the message is not random. A closing curly quote, ”, is three bytes in UTF-8 — and the last of them is 0x9d. cp1252 assigns no character to that byte:

>>> 'a “quote”'.encode('utf-8').hex(' ')
'61 20 e2 80 9c 71 75 6f 74 65 e2 80 9d'
>>> 'a “quote”'.encode('utf-8').decode('cp1252')
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 12: character maps to <undefined>

cp1252 leaves exactly five bytes undefined — 0x81, 0x8d, 0x8f, 0x90, and 0x9d, checked against Python's own codec table.

Any UTF-8 note containing one of them crashes an old Windows reader on a cp1252 install. Every other UTF-8 byte decodes — into the wrong characters, which is the mojibake our byte-level guide to plain-text encoding explains.

The full title of that question is the error message itself: "UnicodeDecodeError: 'charmap' codec can't decode byte X in position Y: character maps to ". When we checked, it showed about two million views.⁸

It is the bug PEP 597 described in 2019 — the one that hides on the developer's own machine: "Even Python experts may assume that the default encoding is UTF-8. This creates bugs that only happen on Windows".⁹ Python 3.15 retires that bug for every script that reads UTF-8 notes without naming an encoding. For a folder already written in UTF-8, that is the whole effect of the upgrade — and it is a good one.

The error it adds: notes an older script wrote in the code page

The failure moves to files written before the upgrade. A pre-3.15 script on Windows that wrote notes without encoding= saved them in the ANSI code page. Read those files on 3.15 and the UTF-8 decoder meets bytes it cannot accept. The result is a loud UnicodeDecodeError — not silent garbling — because the default error handler stays strict.

Here is that failure reproduced. The file is a short note saved as cp1252 — what an older Windows script would have written — and the reader is CPython 3.15.0:

# r1.py: a note saved by an older Windows script, read on 3.15
import sys, locale
print(sys.version.split()[0], sys.flags.utf8_mode, locale.getencoding())
open('old.md', 'wb').write('café – naïve “quote”\n'.encode('cp1252'))

open('old.md').read()
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3:
#   invalid continuation byte

open('old.md', encoding='cp1252').read()
# 'café – naïve “quote”\n'

The bytes are intact. Named correctly, the file reads back exactly.

The é in café is the single byte 0xe9 in cp1252. UTF-8 treats 0xe9 as the start of a three-byte character — and rejects what follows.

The loudness comes from a deliberate choice. Python's UTF-8 mode documentation says the mode keeps "the strict error handler by default so that attempting to open a binary file in text mode is likely to raise an exception rather than producing nonsense data."¹⁰ A crash on the first bad note is the better failure — nothing has been rewritten yet.

The quiet failure exists too, and PEP 686 warns about it plainly: "So this change mostly affects Windows users. When a Python program depends on the default encoding, this change may cause UnicodeError, mojibake, or even silent data corruption. So this change should be announced loudly."⁵ Silent corruption needs help: a script that passes errors="replace" or errors="ignore" and then writes the result back.

There is a second route to the same failure — two Pythons. If a scheduled task still runs an older interpreter while your shell runs 3.15, one writes the code page and the other expects UTF-8.

Java described the same hazard when JDK 18 made UTF-8 its default. JEP 400's authors, Alan Bateman and Naoto Sato, wrote: "If the default charset differs between the JDK of the first application and the JDK of the second application then the resulting text may be silently corrupted or incomplete".¹¹

Ruby made the same move first: Ruby 3.0's release notes record "Changed default for Encoding.default_external to UTF-8 on Windows".¹² PEP 686 notes one difference that helps here: "Both Ruby and Java have an option for backward compatibility. They don’t provide any warning like PEP 597’s EncodingWarning in Python for use of the default encoding."⁵ Python gives you a tool to find the calls. The audit below uses it.

The naive fixes, and why they fail

Two fixes look obvious, and both go wrong. Setting PYTHONUTF8=0 everywhere restores the old behaviour and keeps the original Windows bug alive. Adding encoding="utf-8" to every call with a linter's autofix recreates the new failure on purpose, because it pins UTF-8 onto code that still reads cp1252 files.

Turning UTF-8 mode off for good. The release notes offer the switch: "To retain the previous behaviour, Python’s UTF-8 mode may be disabled with the PYTHONUTF8=0 environment variable or the -X utf8=0 command-line option."¹ It is a real escape hatch. On a cp1252 Windows install it also brings back the 0x9d crash on UTF-8 notes — the bug the change was made to end. Use it for a day while you audit, not as the fix.

Pinning UTF-8 everywhere in one pass. The linter route is tempting because it is mechanical. Ruff's unspecified-encoding rule (PLW1514) — "Derived from the Pylint linter", its page says — finds open() calls without an encoding. It offers to add encoding="utf-8".¹³ Its own documentation labels that fix: "This fix is always unsafe and may change the program's behavior."¹³

The warning is correct. A script that reads old cp1252 notes and gets encoding="utf-8" added now fails on every Python version, not only 3.15. The rule is also in preview, so it needs the --preview flag to run at all. And the same page still says "in reality, the default is locale-specific", which stopped being true on 3.15.¹³

The rule still has value as a list. A 2026 study of OS-dependent bugs found "that only Ruff provides a rule that directly addresses potential portability concerns, namely the unspecified-encoding check".¹⁴

Treat its list as a start, not as complete.

In our run, Ruff 0.17.0 flagged open("a.md") and Path("a.md").read_text() in a test file. It passed this four-line tag fixer, which reads and rewrites every note without naming an encoding:

# tagfix.py
from pathlib import Path
for p in Path("vault").rglob("*.md"):
    text = p.read_text()
    p.write_text(text.replace("#todo", "#task"))

$ ruff check --preview --select PLW1514 tagfix.py
All checks passed!

Python's own runtime warning caught both calls in that script. A linter reads code; the warning watches calls as they happen — which is why the audit below leans on it.

How a text-mode call picks its encoding now

Every text-mode call that omits encoding= follows one short decision path. UTF-8 mode is on by default in 3.15 — so the answer is UTF-8, with strict errors. Only when the mode is switched off does Python ask the locale, and on Windows the locale answers with the ANSI code page.

open(path)  /  Path.read_text()  /  Path.write_text()   (no encoding=)
 |
 +-- UTF-8 mode on  (the default in 3.15) ............ UTF-8, strict errors
 |
 +-- UTF-8 mode off (PYTHONUTF8=0 or -X utf8=0)
      |
      +-- locale.getencoding()
           +-- Windows ............................... ANSI code page (e.g. cp1252)
           +-- Linux, macOS with a UTF-8 locale ...... UTF-8

open(path, encoding="locale") .... locale encoding, even in UTF-8 mode (3.11+)
open(path, encoding="utf-8") ..... UTF-8 on every version and platform

Two details in that diagram are easy to miss. First, encoding="locale" is not new: "The special encoding='locale' argument uses the current locale encoding, and has been supported since Python 3.10."¹ But PEP 686 records that, inside UTF-8 mode, it fell back to UTF-8 until a fix that shipped in 3.11.⁵ Want the code page on purpose? Use 3.11 or later.

Second, locale.getencoding() "ignores UTF-8 mode", per PEP 686, and "This API was added in Python 3.11."⁵ That makes it the one call that still tells you what the old default was on a given machine — even under 3.15.

The deciding rule for each call is about the file, not the interpreter:

The What's New entry gives the same advice in general form: "This only applies when no encoding argument is given. For best compatibility between versions of Python, ensure that an explicit encoding argument is always provided."¹

The audit, in PEP 686's order

PEP 686 ends with a fix guideline, and its first step is the one people skip: "Disable UTF-8 mode." The second is "Use EncodingWarning (PEP 597) to find every places UTF-8 mode affects." Then pin each call, and finally test with UTF-8 mode on. The order matters: you see the old behaviour while you find the calls.

The wording "every places" is the PEP's own.⁵ The four steps below follow its order — the commands are ours.

Step 1 — run the old behaviour on purpose, briefly

On Windows, run the scripts once with python -X utf8=0 script.py. That keeps them reading and writing the code page while you look — so nothing new lands in a second encoding mid-audit. Treat this as a one-day state.

Step 2 — list every call that relies on the default

Python's io documentation describes the tool: "To find where the default encoding is used, you can enable the -X warn_default_encoding command line option or set the PYTHONWARNDEFAULTENCODING environment variable, which will emit an EncodingWarning when the default encoding is used."¹⁵ It has existed since Python 3.10 and still works in 3.15:

$ python3.15 -X warn_default_encoding tagfix.py
tagfix.py:4: EncodingWarning: 'encoding' argument not specified
  text = p.read_text()
tagfix.py:5: EncodingWarning: 'encoding' argument not specified
  p.write_text(text.replace("#todo", "#task"))

Each warning names a file and a line. The same call with encoding='utf-8' produces no warning. Add -W error::EncodingWarning and the warning becomes an exception — a test run then fails on any call you missed.

The warning only fires on code that actually runs. Exercise the paths that matter: the export, the rename, the backup branch that runs once a month.

Step 3 — pin each call with the table above

Most calls in a note script get encoding="utf-8". Reads of old files get the codec those files were written in. The rare call that must follow the machine's code page gets encoding="locale". The 2026 study's recommendation is the same in five words: "explicitly specify encoding when opening text files".¹⁴

Step 4 — test with UTF-8 mode on

On 3.15 this is the default, so a normal run is the test. To check before you upgrade, Python 3.14 still lets you turn the mode on for one run with python -X utf8 script.py; the 3.14 docs note it "can be enabled or disabled using the -X utf8 command line option and the PYTHONUTF8 environment variable."¹⁶

Check the notes your scripts already wrote

Fixing the code does not fix the files. Notes an older script saved in the code page stay that way until something converts them. An eleven-line scanner that reads every .md file as raw bytes and tries strict UTF-8 lists them in one pass. It reads bytes, so its answer does not depend on which Python you run.

# scan.py: run as  python scan.py <notes-folder>
import pathlib, sys
root = pathlib.Path(sys.argv[1])
for p in sorted(root.rglob("*.md")):
    data = p.read_bytes()
    try:
        data.decode("utf-8")
    except UnicodeDecodeError as e:
        print(f"NOT UTF-8  {p}  (byte {e.start}: {data[e.start:e.start+1]!r})")
        continue
    if data.startswith(b"\xef\xbb\xbf"):
        print(f"UTF-8+BOM  {p}")

On a three-file sample folder it printed two lines and stayed silent about the good file, with identical output on 3.15.0 and 3.12.3:

$ python scan.py vault
UTF-8+BOM  vault/bom.md
NOT UTF-8  vault/sub/old.md  (byte 3: b'\xe9')

The UTF-8+BOM line is not a failure. It flags a file that starts with a byte-order mark, which can hide a frontmatter --- from a strict check. Our encoding guide covers what to do with it.

The NOT UTF-8 line is the one to act on — and the byte it reports is a clue. A failure at byte 0 on b'\xff' or b'\xfe' is a UTF-16 byte-order mark, not a code page; the Unicode FAQ lists FF FE as UTF-16 little-endian.¹⁷ An accented letter a few bytes in, like the 0xe9 above, is the usual code-page pattern.

To learn which code page the writing machine used, run this on that machine. It needs Python 3.11 or later, and it still reports the code page under 3.15 because it ignores UTF-8 mode:

python -c "import locale; print(locale.getencoding())"

Then convert only the files the scanner listed. Commit or back up the folder first. The script also keeps a .bak of each original — and stops before writing if the guessed codec is impossible:

# convert.py: run as  python convert.py cp1252 vault/sub/old.md [more files]
import pathlib, sys

codec, names = sys.argv[1], sys.argv[2:]
for name in names:
    p = pathlib.Path(name)
    data = p.read_bytes()
    try:
        data.decode("utf-8")
        print(f"skip  {p}  (already UTF-8)")
        continue
    except UnicodeDecodeError:
        pass
    text = data.decode(codec)  # strict: stops here if the guess is impossible
    p.with_name(p.name + ".bak").write_bytes(data)
    p.write_text(text, encoding="utf-8", newline="")
    print(f"done  {p}  ({codec} -> UTF-8, original kept as {p.name}.bak)")

newline="" keeps each line ending exactly as it was, so a conversion does not also turn CRLF into LF — the other invisible byte a Windows script changes, covered in our line-endings guide. If you would rather convert from a shell, the encoding guide's iconv step does the same job.

Measurement: one file, three configurations

We ran the same cp1252 note through three configurations on Linux, then set them beside the documented Windows behaviour. On Linux the result never changed — which is the point: 3.15 is a Windows change. The Windows rows come from Python's documentation, not from a run, and are labelled that way.

The 3.15 interpreter was the python-build-standalone release for 3.15.0, which reports Python 3.15.0 (main, Oct 9 2026, 15:24:40). The decode results for the Windows rows follow from the codecs themselves — the Linux rows already show UTF-8 rejecting the file and encoding='cp1252' accepting it.

Scale deserves a calm number. Silva, Farahat, and d'Amorim report in their MSR 2026 paper: "Our cross-platform testing of 500 projects reveals that 11.2% exhibit OS-dependent test failures."¹⁴ In their breakdown, the row "Encoding UTF-8 not default everywhere" covers 2 projects in their tests and 7 in mined issues, against 7 and 27 for path separators.¹⁴

That is open-source Python projects, not note scripts.

It says nothing about how many readers this touches. An older survey points the same way from another angle. PEP 597 found that "Of the 4000 most downloaded packages from PyPI, 489 use non-ASCII characters in their README, and 82 fail to install from source on non-UTF-8 locales due to not specifying an encoding for a non-ASCII file."⁹ That survey ran in an ASCII locale, not on Windows.

Trade-offs and what this audit does not catch

The audit finds calls that run and files that fail strict UTF-8. It does not prove a file is right. A file that decodes as UTF-8 may still carry text mis-decoded and saved earlier — and a strict cp1252 decode accepts many files written in another code page. The Windows behaviour here is documented, not reproduced on Windows.

A wrong codec can still succeed. cp1252 rejects only five byte values. A Cyrillic cp1251 note decoded as cp1252 can convert without an error and come out as accented Latin nonsense. Open a converted file and read a line before you delete the .bak.

Old damage looks valid. If a script once read UTF-8 as cp1252 and wrote the result back, the file now holds well-formed UTF-8 mojibake. The scanner passes it. Repairing that is a separate, by-eye job; the encoding guide explains how that garble forms.

Warnings only see what runs. EncodingWarning reports executed lines. A branch your test never reaches stays silent, so read the code too.

Partial runs leave mixed folders. A script that crashes on note 40 may already have rewritten notes 1 to 39. That is a reason for the commit-first habit, not only for conversions.

We did not run Windows. Our test runs were on Linux, where no non-UTF-8 locale was installed. The Windows rows rest on Python's documentation, PEP 686, and the codec results above. A Windows check before trusting any of this in production is one line: python -c "import locale,sys; print(sys.version, sys.flags.utf8_mode, locale.getencoding())" on 3.14 and 3.15.

Living documents move. The Ruff page quoted above may be updated for 3.15 at any time. The change itself landed in CPython issue gh-133711, opened in May 2025 and closed that September.¹⁸

Frequently Asked Questions

These are the questions readers type after an upgrade, answered for a folder of notes. Each answer stands alone, so the same facts repeat where a question needs them. Answers about Windows describe documented behaviour; the Linux answers were measured on CPython 3.12.3 and 3.15.0 under a UTF-8 locale.

Why does Python 3.15 raise UnicodeDecodeError on notes that opened fine before?

On Windows, older Python read text with the ANSI code page, so notes your scripts saved without encoding= were written in that code page. Python 3.15 reads them as UTF-8 by default and rejects bytes such as 0xe9. Open them with the codec they were written in, or convert them to UTF-8.

Does Python 3.15 fix "UnicodeDecodeError: 'charmap' codec can't decode byte X in position Y: character maps to "?

For scripts that read UTF-8 notes without naming an encoding, yes. That error came from older Python on Windows decoding UTF-8 with the ANSI code page — on a Western install cp1252, which has five undefined bytes, including 0x9d. Python 3.15 decodes those notes as UTF-8, so the error goes away for them.

How do I get the old behaviour back in Python 3.15?

Set PYTHONUTF8=0 in the environment or run python -X utf8=0. Both turn UTF-8 mode off, so calls without encoding= use the locale encoding again. On a cp1252 Windows install that also brings back the old 'charmap' error on UTF-8 notes, so use it as a short stop-gap while you audit.

Does anything change on macOS or Linux?

For file reads under a UTF-8 locale, no. We ran the same script and the same cp1252 note on CPython 3.12.3 and 3.15.0 under C.UTF-8 and got identical results. PEP 686 says the change "mostly affects Windows users". A machine set to a non-UTF-8 locale is the exception worth checking.

Should I add encoding="utf-8" to every open() call?

For new code and for notes you write today, yes — it behaves the same on every version. Do not add it blindly to code that reads old files saved in a code page; that recreates the decode error. Ruff's documentation calls its automatic fix for this rule "always unsafe" for that reason.

How do I find every open() call that relies on the default encoding?

Run your scripts with python -X warn_default_encoding, or set PYTHONWARNDEFAULTENCODING=1. Python prints an EncodingWarning with the file and line for each call that omits encoding=, including Path.read_text() and Path.write_text(). Add -W error::EncodingWarning to make tests fail on any call you missed.

How can I tell which encoding an old note was written in?

Scan the folder by trying strict UTF-8 on each file's bytes and listing failures. Then run python -c "import locale; print(locale.getencoding())" on the Windows machine that wrote them; it reports the code page, such as cp1252. A file starting with bytes FF FE is UTF-16, not a code page.

Before you trust a converted folder in production, open three converted notes and read them — and keep the backup until a week of normal use passes. The change itself is CPython's gh-133711.

A default is a guess made once for every file. An encoding argument is a decision made for each one.

mnmnote.com keeps your notes on your own device in open Markdown — plain text that any program you trust can read.