The Vendor Cached Your Note — and Someone Else Can Tell

When you send a note to a cloud AI, the vendor stores part of it to save on compute. That copy — off your device, on their infrastructure — is what prompt caching is. A peer-reviewed audit found that in seven of seventeen providers tested in 2024, a timing attack could confirm whether your input was already cached by a different user.¹

The performance benefit is real and documented. So is the copy.

What prompt caching actually is

Most people know prompt caching as a billing item: send the same preamble twice, and the second call costs less. The mechanism is a key-value cache of the model's attention state, stored in GPU memory on vendor infrastructure. What the documentation rarely foregrounds is that this cache holds a derivative of your text off your device, subject to a retention policy you cannot clear.

OpenAI enables caching automatically for prompts of 1,024 tokens or longer — roughly 750 words.² A cached prefix remains eligible for reuse for at least 30 minutes, but OpenAI may retain it longer with no stated upper bound.³ For organizations without zero-data-retention configured, the default is 24 hours.⁴ Anthropic's default cache lifetime is 5 minutes, extendable to 1 hour at twice the base input price.⁵ Manual clearing is not currently available on OpenAI.⁶

One implementation detail that surfaces in the academic literature: what is cached is the model's key-value tensor state, not the verbatim prompt text.² OpenAI describes it as "only key/value tensors are cached in GPU-local storage, not the prompts themselves." That is an accurate architectural distinction. It does not, as the research below shows, eliminate the observability surface — timing attacks on the cache state can still reveal meaningful information about the inputs that produced it.

The distinction also matters for what the post about token costs and prompt economics covers: caching is sold as a cost-efficiency feature, and it is one. It is both things at once.

What a 2024 audit of seventeen providers found

A September–October 2024 audit of seventeen production LLM API providers — published at ICML 2025 — found that eight providers used prompt caching and seven of those eight made their cache globally shared across users. In those seven, the cache was not isolated per organization, per account, or per user. It was shared.

The audit is Gu, Li, Kuditipudi, Liang, and Hashimoto, Auditing Prompt Caching in Language Model APIs, PMLR 267:20477–20496, ICML 2025.¹ The tests ran in September and October 2024 — that temporal scope is load-bearing. After responsible disclosure during a 60-day window, at least five providers made changes: disabling global cache sharing across organizations and updating their documentation.⁷ The post-disclosure landscape is not the same as what the audit captured.

Global sharing means the following: if two users at separate organizations send the same prefix to the same model, the second request hits the first user's cached attention state. That hit is faster. And that speed difference is detectable.

The audit also produced a secondary finding: evidence that OpenAI's embedding model is a decoder-only Transformer, "which was previously not publicly known."¹ It is a demonstration that the timing side-channel yields information beyond what vendors disclose — not just about your content, but about the architecture of the system processing it.

Gu et al. close on the policy implication: "Because prompt caching may cause privacy leakage, transparency around the caching policies of API providers is important."¹

How a cache timing attack works

A cache timing attack does not read the cache directly. It measures the latency between sending a request and receiving the first response token. A cache hit — the same input sent before — returns measurably faster than a cold inference. By probing with variants of a suspected input and observing which variants arrive fast, an attacker can infer whether a specific phrase was present in someone else's recent call.

Gu et al. describe the surface: "if the cache is shared across users, an attacker could identify cached prompts from fast API response times to learn information about other users' prompts."¹

Zheng et al.'s InputSnatch (preprint arXiv:2411.18191v2) frames the risk as intrinsic to the optimization: "we identify that these cache mechanisms pose a risk of private input leakage, as the caching can result in observable variations in response times."⁸ Their implementation recovered 62% of partial inputs exactly, 12.5% of complete inputs exactly, and 79.5% of inputs at semantic similarity in their test environment.⁹

That scope caveat is mandatory: their experiments ran against a self-deployed vLLM 0.6.2 instance with GPTCache, running LLaMA-2 70B on 8×A40 GPUs, not against a production commercial API.⁹ The recovery rates for OpenAI or Anthropic's production systems are not what those numbers describe. Network latency, hardware heterogeneity, and provider-side noise all degrade the attack. Zheng et al. do not claim otherwise.

Fahey's CacheProbe (arXiv:2605.30613v1, accepted at the SAGAI workshop, IEEE S&P 2026, per the arXiv comment) approached the problem from a different angle: testing through OpenRouter, a routing layer that sits in front of multiple providers. Fahey's finding: "many implementations of prompt caching are not secure against timing attacks or even basic metadata disclosure."¹⁰

Via OpenRouter's default shared credentials, all three providers under test (Groq, Fireworks, and OpenAI) showed cross-account cache sharing. Via BYOK mode, each provider's isolation was restored.¹¹ The Groq result via OpenRouter showed 100% cache hit rate disclosure in metadata; the Fireworks cross-account result reached statistical significance at p = 4.08×10⁻¹⁵.¹² The routing layer, not just the provider, is a variable in the trust model.

What vendors say — and what can be verified

Both major providers state that cache access is isolated between organizations. OpenAI says caches are not shared between organizations and only members of the same organization can access caches of identical prompts.¹³ Anthropic says caches are isolated between organizations and different organizations never share caches, even if they use identical prompts.⁵ These are the current published positions.

The Gu et al. audit ran against the production state of seventeen providers in September and October 2024, and found that seven had not yet implemented equivalent isolation.¹ After disclosure, at least five corrected their behavior.⁷ The reasonable inference is that those specific providers now have isolation policies — not that the audit result was wrong when they ran it.

Gu et al. are explicit about the practical severity, and about the limits of what they achieved. Detecting that something was cached is not the same as reading it: "we were unable to execute practical prompt extraction attacks."¹ Extraction needs an exact prefix match, so one wrong token causes complete failure, and the attacker's own probes poison later measurements — a cache hit may register because the attacker sent the prompt, not because anyone else did. They do not overstate the reassurance either: "We do not claim that prompt extraction attacks are necessarily impossible."¹ The claim is not that an adversary is currently reading your notes from cache timing. The claim is that the surface is audited, documented, and not a theoretical edge case.

What cannot be claimed is that the surface is externally verifiable by users. There is no API call that returns "your prompt was not cached alongside anyone else's." The vendor's documentation is the only confirmation available — and the audit showed the gap between policy and implementation can be real, and close only after disclosure.

A companion essay on what the AI actually receives beyond what you typed covers the assembly problem: the full context that leaves your device is larger than the message you wrote. Prompt caching is one layer of what happens to that content once it arrives.

The architecture of keeping text local

Text you keep local is text no vendor cached. Not as a mitigation applied on top of the API — as the structural consequence of never sending the content to a remote endpoint.

A model running entirely on your device, processing text that never left it, produces no cache entry on vendor infrastructure. The timing surface does not exist because there is no cache. The retention policy is not relevant because there is no copy. The audit methodology cannot reach data that was never transmitted.

This argument is architectural, not polemical. It does not require believing that vendors are acting in bad faith, or that their current isolation claims are false, or that any of the attack scenarios described above are being actively exploited. It requires only accepting that a copy held off your device is a copy that you cannot audit, cannot clear, and cannot verify has not been accessed.

The question of whether your AI assistant trains on your notes is a related but separate issue: training is a different retention surface than caching, governed by different policies and timescales. Caching is shorter-lived and more narrowly defined. Both surfaces share the same entry condition: text leaves your device.

Zheng et al. name the paradox cleanly: "Our work highlights the security vulnerabilities associated with performance optimizations."¹⁴ The caching feature and the privacy surface are the same feature. There is no version of prompt caching that delivers the latency reduction without creating the copy. You accept both together or you work with text that stays where you put it.

Frequently Asked Questions

What is prompt caching and why do vendors use it? Prompt caching stores the model's key-value attention state after processing a repeated input (such as a long system prompt or a fixed document) so that subsequent calls with the same prefix skip re-processing. The second call is faster and cheaper. Most providers enable it automatically for inputs above 1,024 tokens. The cache entry lives in GPU-local storage on vendor infrastructure, not on your device.²

Is my cached prompt shared with other users? OpenAI and Anthropic both state that caches are isolated between organizations; different organizations cannot access each other's cached states.⁵¹³ A September–October 2024 audit of seventeen providers found that seven had globally shared caches at the time of the audit. After responsible disclosure, at least five made changes to implement isolation.¹⁷ Whether current policies hold is not independently auditable by users.

What is a cache timing attack on a language model API? A cache timing attack measures API response latency to infer whether a given input was already cached. A cache hit (an input matching a prior cached prefix) returns faster than a cache miss. By probing with variants of a target phrase and measuring latency across many requests, an attacker can estimate whether that phrase appeared in another user's recent input. Gu et al. confirm the side channel is statistically detectable against production APIs, while being clear about its limit: detecting that something was cached is not the same as reading it, and they "were unable to execute practical prompt extraction attacks."¹

How long does a vendor retain my prompt cache? Retention varies by provider and account configuration. OpenAI: at least 30 minutes, potentially longer with no stated upper bound.³ For organizations without zero-data-retention enabled: 24 hours by default.⁴ Anthropic: 5 minutes by default, extendable to 1 hour at twice the base input price.⁵ Manual clearing is not available on OpenAI.⁶ These are living-document values; verify against current vendor documentation before relying on a specific number.

Did researchers recover actual text from a production API using this attack? The recovery rates in Zheng et al.'s InputSnatch (62% partial input recovery, 79.5% semantic reconstruction) were measured against a self-deployed vLLM instance running on private hardware, not against a production commercial API.⁹ Network latency and production-system noise significantly degrade the attack. Fahey's CacheProbe demonstrated cross-account cache sharing through OpenRouter's shared credentials on production systems, but the attack surface was the routing layer's credential sharing, not the providers' own infrastructure directly.¹¹

Does using BYOK on a routing layer restore cache isolation? In Fahey's tests, using BYOK mode through OpenRouter restored each provider's per-account cache isolation, while OpenRouter's shared default credentials showed cross-account sharing across all three tested providers.¹¹ BYOK removes the shared credential as an attack surface, but does not change the underlying provider's caching behavior or retention policy.

What does local-first mean for prompt caching specifically? If your notes stay on your own device and are never sent to a remote model endpoint, no cache entry for that text exists on vendor infrastructure. The caching surface only exists when content is transmitted to a cloud API. Keeping text local eliminates that surface entirely: not as a layered security control on top of the API, but as a consequence of never transmitting the content to begin with.

The cache was always there. The performance benefit is real. So is the copy — held off your device, under a retention policy you cannot clear, auditable only by researchers running statistical timing tests against a production API you do not control.

Notes are stored on your own device at mnmnote.com — text you keep local is text no vendor cached.