Your Local Model's Context Window Has a RAM Price — and the Default Already Truncated Your Note

When you run a language model locally, the context window is not a sliding scale — it is a fixed RAM reservation. For a machine with less than 24 GiB of VRAM — which covers most consumer setups — Ollama defaults to 4,096 tokens, or roughly 3,000 words of English prose. Any note longer than that is silently dropped. No error. No ellipsis. Just missing text.

That default is not arbitrary. It is a VRAM-tiered choice, and raising it costs memory you may not have. This post explains the mechanism, the layered configuration knobs, and the honest trade-off between context size and RAM capacity. If you run a local model today, this is the configuration constraint most likely to produce wrong answers without telling you why.

What is a context window, and why does RAM pay for it?

The context window is the total number of tokens a model can hold in view during a single inference pass — the working memory described in our post on the cognitive-capacity metaphor behind context windows. Unlike the dollar-per-token cost of a cloud model¹, the cost of a local context window is denominated in RAM, not currency.

The RAM price comes from the key-value cache. During inference, every token the model attends to generates a key and a value vector that must be held in memory while the forward pass continues. The PagedAttention paper (Kwon et al., arXiv:2309.06180, 2023) characterizes the mechanism directly: "the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically."²

When Ollama loads a model, it reserves memory for the full declared context window — the RAM bill is paid at load, not per query. Ollama's documentation states plainly: "Setting a larger context length will increase the amount of memory required to run a model. Ensure you have enough VRAM available to increase the context length."³ The parallel-request formula makes the scaling explicit: "Required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH."⁴

Run a single model with one parallel request slot and a 4,096-token context, and you pay for 4,096 token-positions of KV cache. Expand to 32,768 tokens and that cost scales 8×. The model weights do not change — only the working-memory reservation does.

What is Ollama's actual default? The VRAM-tiered system

The Ollama FAQ states: "By default, Ollama uses a context window size of 4096 tokens. This can be overridden with the OLLAMA_CONTEXT_LENGTH environment variable."⁵

That sentence is accurate but incomplete. A separate Ollama document, context-length.mdx, reveals a tiered system:³

"Ollama defaults to the following context lengths based on VRAM:

• < 24 GiB VRAM: 4k context
• 24-48 GiB VRAM: 32k context

• = 48 GiB VRAM: 256k context"

For a consumer laptop with 8 GiB of unified memory, or a desktop with a single 16 GiB GPU, Ollama defaults to 4,096 tokens. For a workstation with dual 24 GiB cards (48 GiB total VRAM), the default jumps to 256k. The default is not a single number — it is a function of the hardware Ollama detects at launch.

What this means in practice: if you have a note longer than roughly 3,000 words and you run a 4k-context model against it, the bottom of your note is invisible to the model. The model is not lying — it simply never saw that part of the file.

The layer hierarchy: three knobs, one winner

Ollama exposes three configuration layers for context length. They interact in a specific precedence order, and mixing them up is the most common source of "but I set it" confusion.

Layer 1 — Modelfile num_ctx (default: 2,048)

A Modelfile describes a custom model variant. Its parameter table includes:⁶

"num_ctx | Sets the size of the context window used to generate the next token. (Default: 2048)"

If you create a Modelfile and write PARAMETER num_ctx 4096, any model built from it inherits that value — unless a higher layer overrides it.

Layer 2 — Server-level OLLAMA_CONTEXT_LENGTH (default: VRAM-tiered, 4k for < 24 GiB)

The environment variable OLLAMA_CONTEXT_LENGTH is applied at the Ollama server level, before any Modelfile is consulted. For machines with less than 24 GiB of VRAM, Ollama's tiered default lands at 4,096 — which overrides a Modelfile num_ctx of 2,048. This is why most users see 4,096 in practice even though the Modelfile documentation lists 2,048 as the parameter default. The two numbers come from two different layers.

Layer 3 — API num_ctx per request (overrides both)

When you send a request via the Ollama REST API, you can pass num_ctx in the request body. This overrides both the server default and any Modelfile setting, for that request only.

The hierarchy: API request num_ctx > server-level OLLAMA_CONTEXT_LENGTH > Modelfile num_ctx > Modelfile default (2,048).

For llama.cpp users, the equivalent flag is different. The llama.cpp CLI reference documents it as:⁷

"-c, --ctx-size N | size of the prompt context (default: 0, 0 = loaded from model)"

A value of 0 defers the decision to the model's own metadata — a different default behavior than Ollama's VRAM-tiered system. If your model's embedded context metadata is smaller than your prompt, you need to set -c explicitly.

Silent truncation: the failure mode you cannot see

Neither the Ollama FAQ nor context-length.mdx uses the word "truncation." But the behavior is consistent with the architecture and widely documented in Ollama GitHub issues: if your prompt exceeds the declared context window, Ollama drops the tokens that do not fit. No error is raised at the API level. No ellipsis appears in the output.

This produces a specific failure signature when applied to notes:

1. You send a 6,000-word note to a model running with a 4,096-token context.
2. The model receives approximately the first 3,000 words.
3. The model answers based on that portion alone.
4. The answer is confidently wrong about anything that appeared in the second half.

The insidious part is that the response looks normal. The model does not know it missed anything. You would need to already know the note's content well enough to detect when a fact from the second half is absent — which defeats a core purpose of delegating to a model.

The rough word estimate is based on standard BPE tokenizer behavior for English prose: OpenAI's tiktoken project reports approximately 4 bytes per token on average for English text, which translates to roughly 100 tokens per 75 words. At 4,096 tokens, the effective window is approximately 3,000 words.

The RAM bill: an illustrative scaling table

The formula — RAM ∝ OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH⁴ — produces a predictable cost curve as you increase context length. The table below shows approximate relative KV cache multipliers at common context sizes (model weights are fixed and not included in these relative figures):

Actual VRAM consumption depends on the model's hidden size, number of attention heads, quantization level, and whether Flash Attention is enabled. But the scaling relationship — doubling the context window approximately doubles the KV cache RAM cost — is documented and consistent.

Flash Attention: the relief valve

Ollama includes a partial answer to the RAM-growth problem. From the Ollama FAQ: "Flash Attention is a feature of most modern models that can significantly reduce memory usage as the context size grows."⁸

Flash Attention is an algorithmic optimization that tiles the attention computation to avoid materializing the full attention matrix in memory, reducing peak VRAM usage during the attention computation step. At a given context length, a Flash Attention-enabled model uses less VRAM than a standard attention model would. It does not eliminate the KV cache, and it does not change the fact that extending the context window increases total RAM requirements. The curve flattens; it does not disappear.

Check whether your model checkpoint supports Flash Attention before assuming a 32k context is affordable. For models that do not, the scaling table above is the accurate picture.

The honesty gate: bigger is not simply better

Expanding the context window solves one problem and introduces another. Research on attention degradation in long contexts — examined in our post on the context window as working memory — shows that models are not uniformly attentive across the full window.⁹

Information placed in the middle of a long prompt is reliably less well-attended than information at the beginning or the end. A 32k context window does not give you 32k tokens of uniform recall — it gives you a U-shaped attention curve, strong at the edges, weaker toward the center.

The practical consequence: a focused 2,000-word note in a 4k-context model may produce more accurate answers than a sprawling 20,000-token document dump in a 32k model, if the relevant fact lives in the middle of the long document. RAM cost and attention quality are two separate budgets. You can afford the VRAM and still lose the fact.

Do not expand the context window as a substitute for focusing the prompt. Reserve larger windows for tasks that genuinely require multi-document span, and structure long documents to put the most important material at the beginning or end.

How to set the context window

Option 1 — Per-request (API or compatible tool)

Pass num_ctx in the request body. This overrides both the server default and any Modelfile setting, for that request only:

{
  "model": "llama3",
  "prompt": "Your note content here...",
  "options": {
    "num_ctx": 8192
  }
}

Option 2 — Server environment variable (all models, all requests)

Set OLLAMA_CONTEXT_LENGTH before starting the Ollama server. This becomes the new server-level default for every model and every request, unless overridden at the API level:

OLLAMA_CONTEXT_LENGTH=8192 ollama serve

Export it in your shell profile for a persistent setting.

Option 3 — Modelfile (per-model default)

Create or modify a Modelfile for a specific model, then build the variant:

FROM llama3
PARAMETER num_ctx 8192

ollama create my-llama3-8k -f Modelfile

This sets the default for that model variant without affecting others. Useful if you run distinct models for distinct workloads — a long-document model and a short-chat model with different context defaults.

For most note-related workloads, Option 1 or Option 2 is the path of least resistance. Option 3 is the right choice when you want context defaults baked into named model variants that different applications can invoke by name.

The PagedAttention reference: why KV cache is the variable cost

The academic reference point for KV cache memory management in serving systems is the PagedAttention paper (Kwon et al., arXiv:2309.06180, 2023). Its central observation: "the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size."²

PagedAttention addresses the batch-serving case — multiple concurrent users sharing an inference server. It applies a paged memory scheme, borrowed from operating-system virtual memory, to reduce KV cache fragmentation and allow more efficient sharing of pages across requests. This is the mechanism behind vLLM and similar high-throughput serving frameworks, and is touched on in our post on Apple Silicon inference and KV paging as a hardware feature.¹⁰

For a single-user Ollama setup, PagedAttention is less directly relevant — you are not batching requests across users. But the paper makes the core relationship explicit: the KV cache is the variable-cost component of inference, and context length is its primary driver. Model weights are the fixed cost. Context window is the dial.

Frequently Asked Questions

Why does Ollama silently drop tokens instead of returning an error?

The model's forward pass processes up to num_ctx tokens, and anything beyond that boundary is simply not fed in — there is no separate check at the application layer that raises a warning. If you need to detect truncation before it happens, tokenize the prompt yourself and compare the count against your configured num_ctx. Many model clients and scripting wrappers expose a tokenizer or token-count endpoint you can call before submitting.

My machine has 16 GB of unified memory (Apple M-series). What context length is realistic?

Unified memory is shared between CPU and GPU — 16 GB total means model weights, KV cache, and the OS all compete for the same pool. A 7B model at Q4 quantization uses roughly 4–5 GB for weights, leaving 10–11 GB for everything else. At a 4,096-token context, KV cache overhead is modest. At 32k tokens, it grows substantially and may trigger memory compression or swapping. Test with your specific model by watching ollama ps while incrementally raising num_ctx, and watching whether responses slow or memory pressure climbs.

What is the difference between num_ctx and OLLAMA_CONTEXT_LENGTH?

num_ctx is a model-level parameter — it appears in a Modelfile and can be passed per-request via the API. OLLAMA_CONTEXT_LENGTH is a server-level environment variable that sets the default for all models on that Ollama instance. The server-level default (VRAM-tiered: 4k for < 24 GiB VRAM) overrides the Modelfile num_ctx default of 2,048. An API-level num_ctx overrides both. The hierarchy runs: API request → server env var → Modelfile parameter → Modelfile default.

Does increasing the context window make the model smarter?

No. A larger context window gives the model more text to process but does not change its weights or analytical capability. The attention degradation phenomenon means a larger window can produce less reliable results for facts buried in the middle of a long prompt. Expand the window when the information you need genuinely spans the full document — not as a general quality improvement.

How do I estimate how many tokens a note requires?

A rough approximation for English prose: 100 tokens ≈ 75 words, based on the ~4 bytes per token average for BPE-encoded English text. Count the word length of the note, divide by 0.75, and compare the result against your num_ctx setting. For non-English text or code-heavy notes, tokenization density varies — a code block often tokenizes more densely than prose.

Will llama.cpp automatically pick the right context length?

The llama.cpp CLI defaults to 0, which means "loaded from model."⁷ The value embedded in the model's metadata varies by model. Unlike Ollama, llama.cpp does not apply a VRAM-tiered override — it defers to the model file unless you pass -c explicitly. If the model's embedded context length is smaller than your prompt, you need to set -c manually.

What happens if I run two models at once under Ollama?

Each model instance gets its own KV cache allocation. The RAM formula — OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH — applies per model. Running two models simultaneously approximately doubles the total KV cache cost. Parallel requests within a single model multiply further: "a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation."¹¹

The context window is not a free parameter. Every token-position you declare — whether you use it in a given query or not — is VRAM committed at load time. For a machine with less than 24 GiB of VRAM, the default 4,096 tokens is a conservative floor that fits most consumer hardware. Raising it is straightforward, but the RAM bill is immediate and honest.

Know the default. Know the hierarchy. Know that attention does not scale with context the way cost does.

Your notes stay on your own device with MNMNOTE — local-first markdown, no account required. mnmnote.com

References