How to Know Your Notes Haven't Silently Corrupted

Reference: Bairavasundaram et al., "An Analysis of Data Corruption in the Storage Stack" (USENIX FAST '08) — the first large-scale field study of silent data corruption (1.53 million drives, 41 months).

Bit rot is real: a stored byte can flip with no warning, and a plain-text file does not announce it. You detect silent corruption by hashing every file into a manifest and re-checking it later — a mismatch means the bytes changed. But a hash only detects. Repair needs redundancy.

The durability creed of the own-your-files movement usually stops at two rules: keep plain text, and keep backups. Both are right, and both miss a quieter failure. A file can sit on disk for years, be faithfully copied into every backup, and be subtly wrong the whole time — a flipped bit in a cable transfer, a bad sector, a memory error during a write. Steph Ango's "File over app" argues that "the files you create are more important than the tools you use to create them"¹. This post adds the asterisk: you should also be able to prove those files are still the ones you wrote. That proof is cheap, it needs no app and no account, and this post shows the exact commands.

What is bit rot, and is it actually real?

Bit rot is the silent, undetected corruption of stored data over time. It is not a metaphor. The first large-scale field study of it analyzed corruption across "1.53 million disk drives, over a period of 41 months," classifying three types: "checksum mismatches, identity discrepancies, and parity inconsistencies"². This is measured reality, not paranoia.

That study, by Bairavasundaram and colleagues at USENIX FAST '08, found "more than 400,000 instances of checksum mismatches over the 41-month period"³. CERN ran its own investigation in 2007: a probe that wrote known bit-patterns to disk and read them back, deployed on more than 3,000 nodes, where "about 5 weeks of running on 3000 nodes revealed 500 errors on 100 nodes"⁴. Separately, CERN compared stored checksums against a fresh calculation across 33,700 files and found 22 mismatches — "an error rate of one bad file in 1500 files"⁵. Bernd Panzer-Steindel's summary was blunt: "The error rates are at the 10-7 level, but with complicated patterns"⁶.

The number that circulates as "one corrupted bit in 10^9" is folklore — it does not appear in the CERN primary. What the primary actually reports is the 10^-7 error level and those concrete probe counts. The point survives either way: at the scale of a personal archive kept for decades, silent corruption is not a question of if but of how would you even know.

Why plain text doesn't save you here

Plain text is the right foundation, but it protects against a different failure. Markdown survives format rot: no vendor lock-in, readable in any editor for decades. It does nothing about byte rot. A flipped bit in a .md file is still a valid text file — it quietly says something you never wrote, and no error fires.

This is the trap. Because plain text always opens, corruption in it is less visible than in a binary format that would refuse to load. A flipped bit might turn a 1 into a 0 in a number, mangle one character of a quote, or corrupt a word you rarely reread. The file works. The backup of the file works. You find out years later, if ever. Owning the format (covered in what "plain text" actually means) is necessary but not sufficient — you also need a way to notice when the bytes move underneath you.

Detect it: a checksum manifest you own

The simplest integrity tool is a checksum manifest: a one-line-per-file list of cryptographic hashes. Hash every note today, save the list, and re-run it whenever you want. If any file's hash changed, its bytes changed. This needs nothing but a shell — no app, no account, no service — and it works on any folder of files.

Create the manifest with shasum (or sha256sum on Linux), then verify against it later:

# create a SHA-256 manifest of every markdown file
shasum -a 256 *.md > SHA256SUMS

# later — re-hash and compare against the saved manifest
shasum -c SHA256SUMS

Here is a real run on a three-file folder. First the manifest, then a verify pass while everything is intact:

$ shasum -a 256 *.md > SHA256SUMS
$ shasum -c SHA256SUMS
2024-01-05-standup.md: OK
deep-work.md: OK
sourdough.md: OK

Now simulate bit rot: flip exactly one bit in the middle of one file, leaving its size unchanged (byte 46, 0x6d to 0x6c). Re-verify, and the manifest catches it — the corrupted file reports FAILED and the command exits non-zero:

$ shasum -c SHA256SUMS
2024-01-05-standup.md: OK
deep-work.md: FAILED
sourdough.md: OK
shasum: WARNING: 1 computed checksum did NOT match

Use SHA-256, not the older POSIX cksum (a CRC32). A CRC reliably catches accidental single-bit flips, but it is not cryptographic and collides easily on purpose. SHA-256 catches accidental corruption just as well and cannot be forged, so the same manifest doubles as tamper-evidence. Store the manifest in the folder and off-site — the SHA256SUMS file can rot too.

Detect it at scale: content-addressed repos and filesystem scrubs

Two systems give you integrity checking built in. Git names every object by the hash of its contents, so it can prove the stored bytes still match. ZFS and Btrfs checksum every block on disk and re-verify it on read. Both turn "I hope my archive is fine" into a command you run.

Git is a content-addressed store: the object's filename is the hash of its contents. Its own documentation puts it plainly: "Integrity checking is easy. Bit flips, for example, are easily detected, as the hash of corrupted content does not match its name"⁷. So git fsck walks the object store and re-hashes everything. Keep your notes in a Git repo and you get this for one command:

git fsck --full

On a clean repo it prints nothing and exits zero. Corrupt one object on disk and it names the damage immediately:

$ git fsck --full
error: inflate: data stream error (incorrect data check)
error: corrupt loose object '5822e018c16f2250833e82ea8f36e6e7777f85c2'
error: 5822e018...: object corrupt or missing
missing blob 5822e018c16f2250833e82ea8f36e6e7777f85c2

Git is migrating object naming from SHA-1 to SHA-256, because "on 23 February 2017 the SHAttered attack demonstrated a practical SHA-1 hash collision"⁸ — for accidental bit-rot detection SHA-1 is fine, but the stronger hash matters if you also want tamper-evidence. One level down, ZFS and Btrfs store an end-to-end checksum for every block. Their scrub reads all data and re-checks it: in the OpenZFS docs' words, "periodic scrubs can check data to detect and repair latent media degradation (bit rot) and corruption from other sources"⁹. On ZFS that is zpool scrub tank; on Btrfs, btrfs scrub start /mnt.

flowchart TD
  A[Your note files] --> B[Hash each file<br/>into a manifest]
  B --> C{Re-hash later:<br/>hashes still match?}
  C -->|Yes| D[Verified intact]
  C -->|No| E[Silent corruption<br/>detected]
  E --> F{Do you have<br/>redundancy?}
  F -->|Parity or history| G[Repair the bytes]
  F -->|Only a checksum| H[Detected, not fixable:<br/>restore from backup]

Figure: The integrity loop. You hash your files into a manifest, then re-hash later; matching hashes prove the files are intact, and a mismatch proves silent corruption. Detection stops there — repairing the bytes requires redundancy (parity files or version history); with only a checksum you can detect the damage but must restore the file from a backup.

Detect is not repair: redundancy is what recovers

This is the distinction most bit-rot advice blurs. A checksum, git fsck, and a filesystem scrub all detect corruption. None can rebuild the lost bytes on their own. Recovery requires redundancy you set up in advance: parity files, a mirrored disk, or version history. Detection tells you a file is wrong; only redundancy makes it right again.

The FAST '08 authors put it directly: silent corruptions "cannot be detected or repaired by the disk drive itself," and "detecting and recovering from data corruption requires protection techniques beyond those" the disk provides¹⁰.

For a folder you own, the portable redundancy tool is PAR2. It computes Reed-Solomon parity files that "detect damage in data files and repair them if necessary"¹¹, and you choose how much damage to survive. Create 10% recovery data, and it can rebuild a file even after corruption. Here is a real run: create parity, flip a bit, verify (detects), repair (recovers):

par2 create -r10 notes.par2 deep-work.md   # 10% recovery data
par2 verify  notes.par2                     # after a 1-bit flip
par2 repair  notes.par2                     # rebuild from parity

$ par2 verify notes.par2
Target: "deep-work.md" - damaged. Found 23 of 24 data blocks.
Repair is required.  Repair is possible.

$ par2 repair notes.par2
Repair complete.

The same split explains why a ZFS or Btrfs scrub can self-heal but a lone checksum cannot: the filesystem repairs "if possible, by using the RAID protection in suitably configured pools, or redundant copies"⁹ — no redundancy configured, no repair. And it is exactly why detection belongs upstream of backup: if you copy files without verifying them first, you faithfully back up corruption and overwrite the last good version. Detection tells you which copy to trust. This is the layer beneath the 3-2-1 backup rule — backups give you copies; checksums tell you the copy is worth keeping.

The five tools sort cleanly into detectors and repairers. Read this table as a checklist: pick at least one detector, and pair it with at least one repairer.

* Git can restore a bad object from another clone or remote, but git fsck itself only detects.

A routine you can actually run

A practical integrity routine for a notes folder has four moves: hash, store the manifest redundantly, re-verify on a schedule, and keep enough redundancy to repair. It takes minutes to set up and turns durability from a hope into something you can prove on demand.

1. Baseline. shasum -a 256 *.md > SHA256SUMS (or track the folder in Git — git fsck gives you the same guarantee plus history).
2. Protect the manifest. Store SHA256SUMS alongside the notes and in an off-site copy, so a rotted manifest can itself be caught.
3. Verify on a cadence. Run shasum -c SHA256SUMS, git fsck, or a filesystem scrub monthly — before, not after, your backup runs.
4. Keep redundancy for repair. PAR2 parity files, a mirrored/ZFS pool, or version history — so a detected flip is fixable, not just visible.

Why owning your files makes this possible

The reason you can do any of this is that the files are yours. Every tool here — shasum, git fsck, a ZFS scrub, PAR2 — operates directly on the bytes. It runs offline, needs no permission, and answers a yes-or-no question about your own data. An opaque vendor database gives you no such lever: you cannot hash what you cannot reach.

That is the quiet case for local-first, own-your-data storage, made concrete (the broader argument lives in software moving back to your device). "File over app" says your files should outlive the tool¹; integrity verification is what lets you confirm they did. Ango's fuller line is worth keeping in view: "In the fullness of time, the files you create are more important than the tools you use to create them. Apps are ephemeral, but your files have a chance to last"¹. A file you can hash is a file whose survival you can audit. A row in a database you will never see is a file you can only trust.

Frequently Asked Questions

What is bit rot, and is it real? Bit rot is the silent corruption of stored data over time — a bit flips on disk, in transfer, or in memory, with no error raised. It is measured: a study of 1.53 million drives over 41 months found more than 400,000 checksum mismatches²³, and CERN's own probe surfaced hundreds of silent errors⁴.

Can a checksum repair a corrupted file? No. A checksum (SHA-256), git fsck, and a filesystem scrub only detect that bytes changed. Repair requires redundancy you set up beforehand: PAR2 parity files, a mirrored or ZFS pool, or version history. The FAST '08 authors note detection and recovery are separate problems needing separate protections¹⁰.

How do I know my notes haven't corrupted? Hash them into a manifest — shasum -a 256 *.md > SHA256SUMS — and re-run shasum -c SHA256SUMS later. Any file whose hash changed reports FAILED. If you keep notes in Git, git fsck --full gives the same guarantee, because objects are named by their content hash⁷.

Does plain text protect against bit rot? Only against format rot, not byte rot. Plain text stays readable in any editor for decades, but a flipped bit in a .md file is still a valid text file — it opens fine and shows something you never wrote. Plain text plus a checksum manifest covers both failures.

Should I use SHA-256 or CRC (cksum) for a manifest? Use SHA-256. A CRC32 (cksum) reliably catches accidental single-bit flips but is not cryptographic and collides easily, so it cannot detect deliberate tampering. SHA-256 catches accidental corruption equally well and doubles as tamper-evidence at no extra cost.

How does ZFS or Btrfs protect against bit rot? Both store an end-to-end checksum for every block and re-verify on read. A periodic scrub re-checks all data "to detect and repair latent media degradation (bit rot)"⁹ — and repairs it automatically only when the pool has redundancy (a mirror or RAID). Without redundancy the scrub detects but cannot heal.

How often should I verify my files? Monthly is a reasonable cadence for a personal archive, and always before a backup run so you never copy corruption over your last good version. Automate it: a scheduled shasum -c, git fsck, or zpool scrub needs no attention until something actually fails.

Silent corruption is the one failure that does not knock. The fix is not to hope your bytes are fine — it is to hold the one number that tells you when they are not.

If you keep notes as files you own, you can hash them, verify them, and prove they are intact — try it in mnmnote.com.