The Append-Only Log File That Never Rewrites History

Reference: POSIX / Linux open(2) — the O_APPEND flag — man-pages project (public domain / GPL) · C / POSIX

An append-only log file is a plain-text file you only ever add lines to — never editing or deleting what is already there. It buys crash-safety and auditability almost for free: because each record is written to the end and never overwritten, a power loss can corrupt only the final line, leaving every earlier record intact. That single design choice is why databases and message queues are built on logs.

This is the byte-layer version of a discipline you may already run by hand. The human side — a dated note you append to all day and never reorganize — is its own good habit.¹ Letting an AI agent add lines instead of rewriting your file is another.² Here we go one level down: what the operating system actually guarantees when you append, where those guarantees stop, and how to design a format that survives a crash.

Why never rewriting history is a safety feature

Rewriting a file in place means seeking to some offset and overwriting existing bytes. If the machine loses power halfway through, you get a file that is neither the old version nor the new one — a corruption buried in the middle of your data, with no marker telling you where it starts. Append-only deletes that entire failure mode.

When you only add to the end, the bytes that were already committed are never touched again. A crash can leave a half-written record at the very tail of the file, but the reader can detect and drop that one incomplete line — everything before it is exactly as it was. You trade in-place mutation for an ever-growing file, and in exchange every past record becomes immutable history. The file stops being a mutable blob and becomes an event log: not "what is the current state" but "here is everything that happened, in order."

That immutability is also what makes the file auditable. Because no line is ever edited, the file is the history — you can diff it, replay it, or hand it to a reviewer, and nothing can have been silently changed underneath.

How an atomic append actually works

The mechanism is one operating-system flag. When you open a file with O_APPEND, the kernel guarantees that finding the end of the file and writing your bytes happen together: "The modification of the file offset and the write operation are performed as a single atomic step."³ Two processes appending to the same file can never land on top of each other — each write goes cleanly after whatever was there when it started.

In Python, that flag is what the "a" open mode sets. A minimal durable append looks like this:

import json, os

def append(path, record):
    line = json.dumps(record, separators=(",", ":")) + "\n"
    # O_APPEND: the seek-to-end and the write are one atomic step
    fd = os.open(path, os.O_WRONLY | os.O_APPEND | os.O_CREAT, 0o644)
    try:
        os.write(fd, line.encode("utf-8"))
        os.fsync(fd)          # without this, the bytes may still sit in cache
    finally:
        os.close(fd)

Now the honest part, because the guarantee is narrower than it sounds. The atomic step covers the offset adjustment plus one write() call — not a durability promise, and not an unlimited size. The closest sibling guarantee spells out the limit: on pipes, "POSIX.1 says that writes of less than PIPE_BUF bytes must be atomic ... Writes of more than PIPE_BUF bytes may be nonatomic: the kernel may interleave the data with data written by other processes."⁴ POSIX requires PIPE_BUF to be at least 512 bytes, and on Linux it is 4096 bytes.⁴ Keep each record comfortably under that, write it in a single call, and you stay inside the well-defined zone.

There is one place the atomicity breaks outright: networked storage. The manual is blunt — "O_APPEND may lead to corrupted files on NFS filesystems if more than one process appends data to a file at once ... NFS does not support appending to a file, so the client kernel has to simulate it, which can't be done without a race condition."⁵ An append-only log is a local-disk design. Put it on a network share with concurrent writers and the core guarantee is gone.

The write path, from record to durable byte

Appending safely is two guarantees stacked, not one. O_APPEND decides where your bytes land — atomically at the end. fsync decides when they are actually on the disk. Skip the second and a crash can still lose the record you thought you wrote, because a successful write() only moves bytes into the operating system's cache. The diagram below is the whole path.

flowchart TD
  A[New record] --> B[Serialize to<br/>one text line]
  B --> C[write in<br/>O_APPEND mode]
  C --> D[Seek-to-end plus write:<br/>one atomic step]
  D --> E[Bytes in<br/>OS page cache]
  E --> F[fsync flushes<br/>to disk]
  F --> G[Record is<br/>durable]
  E -. crash before fsync .-> H[Torn tail<br/>record dropped]
  G --> I[Earlier records<br/>stay intact]
  H --> I

Figure: The append write path. A record is serialized to one line and written in O_APPEND mode, where seek-to-end and write are a single atomic step; the bytes first land in the OS page cache, and fsync flushes them to durable storage. If the machine crashes before fsync, only the unflushed tail record is lost — every earlier record stays intact.

The failure branch is the point. A crash between the write() and the fsync() can drop the last record, but it can never damage an earlier one, because those bytes were finalized on a previous append. Recovery is not a repair; it is just "read until the tail stops parsing." That is the durability model an append-only log gives you, and it is far simpler than the in-place equivalent.

One line, one record: the JSONL format and crash recovery

The format that makes this practical is line-delimited: one record per line, one newline between them. JSON Lines — "also called newline-delimited JSON" — is the common spelling, and its own spec notes it is "a great format for log files." It has exactly three requirements: UTF-8 encoding, each line a valid JSON value, and a line terminator of \n.⁶ A file looks like this:

{"t":"2026-07-22T09:01:04Z","op":"add","id":"n-4f2","text":"call the vet"}
{"t":"2026-07-22T09:03:11Z","op":"done","id":"n-4f2"}
{"t":"2026-07-22T09:07:52Z","op":"add","id":"n-5a9","text":"draft the memo"}

The newline is not decoration — it is the record boundary that makes crash recovery trivial. Because every complete record ends in \n, a torn final write is self-identifying: it is the one line without a terminator, or the one that fails to parse. Reading the log means taking every line up to that point and stopping:

def read_log(path):
    records = []
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            if not line.endswith("\n"):
                break            # no terminator: a torn tail write, drop it
            try:
                records.append(json.loads(line))
            except json.JSONDecodeError:
                break            # unparseable tail: stop, earlier lines are valid
    return records

The log grows forever, which sounds like a problem until you notice the fix is also append-only. To compact — collapse the history down to current state — you never edit in place. You write a fresh file and swap it atomically:

def compact(path, live_state):
    tmp = path + ".compacting"
    with open(tmp, "w", encoding="utf-8") as f:
        for record in live_state:
            f.write(json.dumps(record, separators=(",", ":")) + "\n")
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, path)   # atomic rename: readers see the old OR new file, never a half one

os.replace maps to an atomic rename, so a reader always opens a complete file — the old one or the new one, never a half-written mix. The append-only discipline holds even when you shrink the log.

Prior art: your database already appends

If this feels like reinventing a database, that is because databases reinvented it first — append-then-checkpoint is the standard trick for crash-safe writes. SQLite's Write-Ahead Log is the clearest example: "The original content is preserved in the database file and the changes are appended into a separate WAL file," and "Writers merely append new content to the end of the WAL file."⁷ A transaction commits by appending one record: "A COMMIT occurs when a special record indicating a commit is appended to the WAL. Thus a COMMIT can happen without ever writing to the original database."⁷ The main file is only updated later, in a checkpoint — by default "when the WAL file reaches a threshold size of 1000 pages."⁷

Message queues take the idea further and make the log the entire storage engine. Apache Kafka's design notes explain the payoff directly: "Intuitively a persistent queue could be built on simple reads and appends to files ... This structure has the advantage that all operations are O(1) and reads do not block writes or each other."⁸ Compare that to a B-tree, where "Btree operations are O(log N)."⁸ Kafka keeps writes cheap because "the log allows serial appends which always go to the last file."⁸ The append-only log is not a poor cousin of a database — it is the primitive the durable ones are built on.

What to steal from the append-only log

You do not need Kafka to use the pattern; you need four rules. First, open with O_APPEND so concurrent writers never overwrite each other. Second, one record per line, so a torn write damages exactly one line and the boundary is self-evident. Third, fsync when durability matters, because a returned write() is not yet on disk — and reach for fdatasync when you only need the data flushed, since it "does not flush modified metadata unless that metadata is needed."⁹ Fourth, compact by rewriting a new file and renaming, never by editing in place.

The reusable insight underneath all four is that immutability is cheaper than mutability at the storage layer. An in-place format has to defend against every partial write in the middle of the file; an append-only format only has to check its own tail. Fewer states means fewer bugs. That is why the pattern shows up everywhere from Git's object store to filesystem journals to the humble application log.

Verdict: when an append-only file is the right call

Reach for an append-only log when your data is naturally a sequence of events — notes, edits, actions, measurements — and you value being able to replay or audit the whole history. It is a superb fit for local-first tools, where the file is the source of truth and you cannot assume a database server is running. It is the wrong tool when you need random in-place updates to huge records, heavy concurrent writes across a network share, or point queries into billions of rows — that is what indexes exist for.

But be honest about the limits, because they are documented, not theoretical. O_APPEND atomicity covers a single write on a local filesystem and breaks on NFS.⁵ A write() that returns success "does not make any guarantee that data has been committed to disk."¹⁰ Durability is a second, separate call: fsync flushes data "so that all changed information can be retrieved even if the system crashes or is rebooted."⁹ Respect those two boundaries and an append-only file gives you a format that a crash can dent but never quietly corrupt.

Frequently Asked Questions

Is O_APPEND atomic? Yes, within limits. When a file is opened with O_APPEND, "the modification of the file offset and the write operation are performed as a single atomic step,"³ so concurrent writers never overwrite each other. That covers one write() on a local filesystem. It is not a durability guarantee, and it "may lead to corrupted files on NFS filesystems" under concurrent appends.⁵

Does a successful write() mean my data is on disk? No. "A successful return from write() does not make any guarantee that data has been committed to disk."¹⁰ The bytes usually sit in the operating system's page cache until it flushes them. To guarantee the record survives a crash, call fsync (or fdatasync) after writing, then check that it returned success before treating the record as durable.

What is the difference between fsync and fdatasync? fsync flushes all modified data and metadata for the file. fdatasync "is similar to fsync(), but does not flush modified metadata unless that metadata is needed in order to allow a subsequent data retrieval to be correctly handled."⁹ For an append-only log, where the file only grows, fdatasync is often enough and slightly faster, because timestamp metadata does not affect reading the data back.

What is JSONL / NDJSON? JSON Lines is a text format "also called newline-delimited JSON" with three requirements: UTF-8 encoding, each line a valid JSON value, and a line terminator of \n.⁶ One record per line makes it "a great format for log files": you can append with a single write, stream it line by line, and detect a torn final record because it lacks a terminator or fails to parse.

Why are append-only logs considered crash-safe? Because you never overwrite committed bytes. A crash can leave a half-written record only at the very end of the file, and the reader drops that one incomplete line — every earlier record is untouched. In-place editing has no such property: a power loss mid-write can corrupt data anywhere in the file, with no marker showing where the damage is.

How does SQLite's WAL mode relate to this? SQLite's Write-Ahead Log is the same pattern inside a database. "The original content is preserved in the database file and the changes are appended into a separate WAL file,"⁷ and a commit is just an appended record. The main database file is updated later during a checkpoint, which runs by default "when the WAL file reaches a threshold size of 1000 pages."⁷

How do I stop an append-only log from growing forever? Compact it, but never in place. Write the current state to a new file, fsync it, and atomically rename it over the old one, so a reader always sees a complete file — the old one or the new one. This keeps the append-only guarantee intact even while shrinking the log, and it mirrors how Kafka rolls "serial appends which always go to the last file."⁸

Is an append-only file slower than a database? For its natural workload, no — it is faster. Sequential appends are O(1), while B-tree updates are O(log N), which is why Kafka notes that a log-structured store means "all operations are O(1) and reads do not block writes or each other."⁸ The trade-off is querying: without an index you scan, so append-only logs suit event and time-ordered data, not random point lookups into huge datasets.

References

The safest place to put history is a file that only knows how to grow.

If you want notes that behave like this — plain text you own, stored locally, that a crash can dent but never quietly rewrite — that is the idea behind mnmnote.com.