Tell the Crawlers to Leave Your Notes Alone: robots.txt, ai.txt, and the AI-Preferences Standard

You can attach a machine-readable do-not-train boundary to notes you publish — a line in robots.txt, an HTTP header, a file at a known address. But it is a request a well-behaved crawler honors, not a wall. The only true enforcement is authentication, a paywall, or not publishing at all.

For twenty-five years, robots.txt was a convention — a widely honored agreement with no formal standing. That changed in September 2022, when the IETF published RFC 9309 and made the Robots Exclusion Protocol a real Standards-Track specification, extending the method Martijn Koster first defined in 1994.¹ So the file you would use to address AI crawlers is now genuine infrastructure. What it is not is a lock. The same RFC says plainly that its rules "are not a form of access authorization."¹ The Electronic Frontier Foundation's Thorin Klosowski put the reader's real situation in one sentence: "There's no technical requirement that a bot obey your requests."² Everything that follows — the crawler tokens, the emerging AIPREF directive, the experimental ai.txt file — sits inside that one honest limit. You are posting a sign, not building a fence.

What the stale advice tells you to do

The common instruction is a robots.txt recipe: name the training crawlers and disallow them. GPTBot, ClaudeBot, CCBot, and Google-Extended each publish a user-agent token for exactly this purpose. The advice is not wrong, and it is not nothing. It is incomplete, because it stops at a verb the standard never promised: block.

Here is the recipe as it usually appears, and it does real work:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Each of these companies publishes the token on purpose. OpenAI documents GPTBot/1.4 and writes that "GPTBot is used to make our generative AI foundation models more useful and safe."³ Anthropic tells site owners they can disallow ClaudeBot, which "helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training."⁴ Common Crawl, the open-web corpus many models are built from, identifies its agent as CCBot/2.0.⁵ Google separates training from search with a dedicated Google-Extended token.⁶ Publishing an opt-out token is a cooperative act, and it deserves credit. The problem is never that these tokens exist. The problem is the word wrapped around them.

Why "block the crawlers" is the wrong verb

A robots.txt directive does not block a crawler; it asks one to stay away. RFC 9309, the ratified standard, says as much in its own security section: the protocol "is not a substitute for valid content security measures," and its rules "are not a form of access authorization."¹ A good crawler honors the request. A bad one reads past it.

Two consequences follow, and both are easy to miss. The first is retroactive: a disallow only governs future crawls. Klosowski is blunt about it — "Keep in mind that this only applies to future scraping. If Google or OpenAI already have data from your site, they will not remove it."² A sign posted today says nothing about the copies already taken.

The second is structural. The people building the successor standard say the same thing about their own work. The IETF's AI Preferences charter lists "Technical enforcement of preferences" as explicitly out of scope⁷ — the working group is deliberately not building a lock. As its co-chairs Mark Nottingham and Suresh Krishnan write, "Likewise, preferences are just that: they aren’t technically enforced."⁸ The honest verb was never "block." It was always "ask" — and, as Klosowski notes, "many crawling services respect the request."²

The emerging standard is real, and still moving

A newer answer is taking shape, and it is worth watching without leaning on. In January 2025 the IETF chartered the AI Preferences Working Group, or AIPREF, to standardize how content expresses its training preferences.⁸ Its draft defines a Content-Usage directive for robots.txt and HTTP headers. It is unratified, and it is still moving.

The draft, draft-ietf-aipref-attach-04, is intended for the Standards Track and even proposes to update RFC 9309 if approved; it was published on 28 October 2025 and expired on 1 May 2026 without a successor revision.⁹ It lets a publisher attach a usage preference as both an HTTP header and a robots.txt directive:

User-Agent: *
Allow: /
Disallow: /never/
Content-Usage: train-ai=n
Content-Usage: /ai-ok/ train-ai=y

Treat that syntax as provisional, because it has already changed. In the mid-2025 revision the opt-out token was Content-Usage: ai=n; by the late-2025 revision it had become Content-Usage: train-ai=n.⁹ A standard whose core keyword shifted in a matter of months is not one to hard-code your assumptions against.

A separate proposal, ai.txt, would place a policy file at /.well-known/ai.txt. It is weaker in status still — an Independent Informational draft, not AIPREF working-group output, published 12 June 2026 and expiring 14 December 2026.¹⁰ Its author, K. Cardillo, is candid about what it does: the file "is advisory," and "Compliance is expected in good faith but is not enforced by the file itself."¹⁰ Every rung of this ladder is built from the same material: a preference you declare, honored by those who choose to.

What to do with notes you actually publish

Do three things, in descending order of how much they actually hold. Attach today's signals: the named crawler tokens, and a Content-Usage line if you want to be early. Watch AIPREF without betting your privacy on it. Then draw the real boundary — anything that must never be trained on does not belong on a public page.

The reassuring detail first: opting out of AI training does not cost you search visibility, as long as you use the right token. Google's documentation states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."⁶ The training opt-out and the search crawler are separate controls, so you can decline one without losing the other.

With that fear removed, the practice is short:

1. Post the named-crawler disallows in the robots.txt at your site's root — GPTBot, ClaudeBot, CCBot, Google-Extended, and any others you find on a vendor's own page.
2. Add a Content-Usage line if you want to signal to AIPREF-aware crawlers early, knowing the syntax may change under you.
3. Skip or sandbox ai.txt until it graduates past an expiring individual draft.
4. Keep the truly private off the public web — behind authentication, or simply unpublished. That is the only rung that enforces.
5. Store the boundary where you store the notes. robots.txt, a Content-Usage header, an ai.txt — each is a plain-text file you own, sitting beside the plain-text notes whose preference it carries.

That last point is the durable one. When you publish notes written in a local-first, plain-Markdown tool, the file stays yours and so does the boundary you attach to it. If you publish your notes as a static site, the robots.txt and any Content-Usage header live right there in the same directory you control. It is the same ownership argument this blog makes about your data in general, carried out to the edge of what you choose to share. The inbound side, what an assistant you paste into may train on, is a separate question, covered in whether your AI assistant trains on your notes.

Frequently asked questions

Short answers to what people actually type when they want AI companies to leave their published writing alone. Each one holds the same honest line as the rest of this piece: these are signals a cooperating crawler honors, dated to a standard that is still being written, and none of them is a lock.

How do I stop AI from training on my website? You cannot force a stop, but you can post a boundary most crawlers respect. In robots.txt, disallow the named training agents — GPTBot, ClaudeBot, CCBot, Google-Extended. Optionally add a Content-Usage line from the emerging AIPREF draft. All of it is voluntary; only auth, a paywall, or not publishing enforces.

How do I block GPTBot in robots.txt? Add two lines to the robots.txt at your site's root: User-agent: GPTBot then Disallow: /. OpenAI documents the token as GPTBot/1.4 and states that disallowing it "indicates a site's content should not be used in training generative AI foundation models." It is a request the crawler is documented to honor, not a firewall.

Does robots.txt block AI crawlers? Not in the enforcement sense. It requests; it does not compel. As the EFF's Thorin Klosowski puts it, "There's no technical requirement that a bot obey your requests." Reputable AI crawlers publish tokens and honor disallows; an unnamed scraper can ignore the file entirely. The word to use is "ask," not "block."

How do I opt out of AI training data? Layer the tools. Disallow the named crawlers in robots.txt, add the AIPREF Content-Usage directive if you want to signal early, and treat an ai.txt file as experimental. Each is a preference a cooperating company honors. For anything you truly cannot risk being trained on, the only real control is to keep it off the public web.

Does blocking AI crawlers hurt my Google or SEO ranking? No, if you use the right token. Google-Extended is a separate control from Googlebot; opting out of Gemini training does not remove you from search. Google's own documentation states Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Disallow it without fear for your rankings.

What is ai.txt? A proposed declaration file that would live at /.well-known/ai.txt and state your AI-usage policy. As of mid-2026 it is only an Independent Informational IETF draft — not a ratified standard, and separate from the AIPREF Working Group. Its own author writes that the file is advisory and expects compliance "in good faith." The draft expires in December 2026.

How do I block ClaudeBot and CCBot? Two more disallow blocks in the same robots.txt. Use User-agent: ClaudeBot with Disallow: /, and User-agent: CCBot with Disallow: /. Anthropic documents ClaudeBot as collecting web content that could contribute to model training; Common Crawl identifies its agent as CCBot/2.0 and feeds many downstream models. Both honor the disallow when you name them.

A published note is a note you have handed to the open web, crawlers and all. What you still control is the sign you post beside it and the honesty with which you read that sign — a request the well-behaved will honor, a standard still being written, and one line, auth or silence, that is the only thing a determined reader cannot cross. Own the file. Own the boundary. Know which one actually holds.

If you want the notes and the boundary to live in the same place you control, mnmnote.com keeps your writing as plain Markdown on your own device and lets you publish only what you choose.