Ask Your Notes a Question on a Local Model and You Wait 14 Seconds — the Search Took 5 Milliseconds

If you have set up a local AI pipeline to query your notes and the first question takes ten or fifteen seconds, you are probably optimizing the wrong thing. The vector index is fast — 5 milliseconds fast. The wait is two language model calls in sequence, preceded by a cold model load that can cost more than six seconds on its own.

This post measures where the time goes, explains the two Ollama defaults that control most of it, and proposes four fixes in the order that will buy back the most seconds. The numbers come from one practitioner's measured run on a mid-range laptop running a small 3B model — treat them as a concrete example, not a universal benchmark, because laptop LLM timings move significantly between runs. The generalisable finding is the structure: cold load, then planning call, then search, then answer call. That structure holds wherever Ollama's defaults hold.

The measurement: where the 14 seconds went

A four-stage pipeline — plan the query, embed the question and search the notes, generate the answer — recorded a p50 total of 14.06 seconds across a run of questions. The search stage, which everyone assumes is the bottleneck, took 4.5 milliseconds. Planning and answering, the two language model calls, together accounted for the overwhelming majority of the wait.¹

The timing breakdown, measured against a small 3B model on a mid-range laptop running Ollama, was recorded by practitioner Arinay on the mindsageapp.pages.dev engineering blog:¹

*Each share is that stage's median divided by the total median. Because medians are independent, the column does not partition to 100%.¹

flowchart LR
    Q[Your question] --> P[LLM call 1<br/>Query planning<br/>6.16 s]
    P --> E[Embed question<br/>32 ms]
    E --> S[Vector search<br/>4.5 ms]
    S --> A[LLM call 2<br/>Answer generation<br/>8.15 s]
    A --> R[Answer]

The four-stage pipeline in order, with p50 durations from the measured run. The two language model calls sit on either side of the search, and with Ollama's default of one request at a time, they run strictly one after the other.

As Arinay summarised: "Retrieval, embedding and search together, was a fifth of one percent. The whole cost was two language model calls in a row, against an Ollama server that handles one request at a time."¹

The variance matters as much as the number. A later rerun of the same pipeline recorded a p50 of 10.48 seconds — a 26% difference from the same test, with no configuration change. Arinay noted: "LLM timings on a laptop move a lot between runs."¹ The 14.06 s figure and the 10.48 s figure are both real measurements from the same setup. Any pipeline that shows you a single number for a local model run is showing you one snapshot.

The cold-load problem: 6.21 seconds before the first token

The single most expensive event in a fresh pipeline run is not question-answering — it is loading the model from disk. On the same mid-range laptop, the first AI call after launch took 6.40 seconds, of which 6.21 seconds was the model loading from disk. The same call, made while the model was already resident in memory, took 224 milliseconds. Time to first token once loaded was approximately 400 milliseconds.¹

That 6.21-second gap is entirely explained by disk I/O, not by the difficulty of the question. It is the cost of a cold start — and it repeats every time the model has been evicted from memory.

By default, Ollama keeps a model in memory for five minutes before unloading it. The official FAQ states: "By default models are kept in memory for 5 minutes before being unloaded."² That five-minute window is why the first question after a gap feels so much slower than subsequent ones. If you ask a second question six minutes after the first, you are paying the cold-load cost again.

The Ollama API exposes a keep_alive parameter per request that overrides the default. Setting it to a longer value — or to -1 to keep the model loaded indefinitely — eliminates the cold-load cost for all subsequent questions in a session:³

curl http://localhost:11434/api/generate -d '{
  "model": "your-model-name",
  "prompt": "Which of my notes mention async workflows?",
  "keep_alive": "30m"
}'

The same parameter is available on the /api/chat endpoint. Setting it in a programmatic pipeline means the model stays resident across the planning call and the answer call — and across multiple questions from the same user session.

You can see how much time loading cost in a given request by reading the response's load_duration field:³

{
  "load_duration": 6338219291,
  "prompt_eval_duration": 1234567890,
  "eval_duration": 2345678901
}

All three values are in nanoseconds. If load_duration is large relative to eval_duration, the dominant cost was the cold start, not the generation itself.

LM Studio uses a different default. Its just-in-time TTL defaults to 60 minutes, and an explicit lms load command holds the model indefinitely.⁴ If you are comparing cold-start behaviour between Ollama and LM Studio, the five-minute vs. sixty-minute default is the first difference to understand.

The serial-call problem: two answers in a row, one at a time

Most local RAG pipelines make two calls to the language model: one to plan or rephrase the query, and one to generate the final answer from the retrieved context. Both calls are made sequentially, to an Ollama server that by default handles exactly one request at a time.

The Ollama FAQ documents this directly: OLLAMA_NUM_PARALLEL defaults to 1.² One request at a time means the planning call must fully complete before the answer-generation call can begin — no overlap, no interleaving. With the mid-range laptop timings above, that serialisation cost was 6.16 seconds (planning) plus 8.15 seconds (answering) — over fourteen seconds of sequential LLM work, with the 4.5-millisecond search sandwiched between them.

If your pipeline does not need a separate planning call — if the user's query is already specific enough to embed directly — removing it cuts one full LLM round-trip. That is the highest-leverage architectural change available.

The companion post on context-window sizing and memory limits goes deeper on OLLAMA_NUM_PARALLEL and what increasing it costs in VRAM — that tradeoff lives there: Your Local Model's Context Window Has a RAM Price.⁵

What the research says about cold starts at scale

The cold-start problem for language models is not a local-setup curiosity. Fu et al.'s 2024 OSDI paper ServerlessLLM measured loading times for production-scale models: OPT-30B loaded in 34 seconds across four GPUs with PyTorch; LLaMA-2-70B in 84 seconds across eight GPUs.⁶ The mid-range laptop's 6.21-second load time for a small 3B model is proportionally consistent with that finding — disk I/O dominates cold starts at every scale.

The same paper analysed an Azure production trace and found that more than 40% of serverless AI functions experienced a cold-start rate above 25%, even with a five-minute keep-alive window.⁶ The five-minute Ollama default sits at the same threshold the Azure trace identified as insufficient for a significant fraction of real usage patterns.

For a personal note vault queried intermittently throughout the day — not on a strict schedule — the five-minute eviction window means cold starts are common. The fix is to extend the window to match how you actually use the pipeline.

Four fixes, in the order you should try them

The bottleneck is not the index. These four changes are ordered by their expected impact on the cold-start and serial-call structure described above.^{1 2}

Fix 1: Stop turning streaming off

Ollama's API streams token-by-token by default. Many RAG pipelines disable streaming because they request structured JSON output from the answer step — and the JSON parser needs the complete response before it can proceed.

The practical fix: stream the answer field, parse it incrementally, and treat the JSON container as complete once the stream closes. Turning streaming off for the answer call means the user waits for every token to be generated before seeing the first word. Streaming restores time-to-first-token feedback even when the full response takes several seconds to complete.

Fix 2: Set keep_alive to stay resident

For a personal pipeline, the right value is the duration of your longest working session. Thirty minutes to two hours is a reasonable starting point for most note-query workflows. Setting it to -1 removes eviction entirely, which is appropriate if the model fits comfortably in memory and the machine is not shared.

# Keep the model loaded for 2 hours
curl http://localhost:11434/api/generate -d '{
  "model": "your-model-name",
  "prompt": "...",
  "keep_alive": "2h"
}'

You can also set the system-level default via the OLLAMA_KEEP_ALIVE environment variable, so every request inherits a longer window without changing individual calls.

Fix 3: Decide whether the planning call is earning its cost

A planning call rewrites the user's question into a form optimised for retrieval. At 6.16 seconds p50 on the mid-range laptop, it costs more than the answer-generation step in absolute terms. Whether it earns that cost depends on the specificity of your users' queries and the density of your note vault.

For a personal vault of a few hundred to a few thousand notes, direct embedding of the user's query often performs comparably to a planned rewrite — especially for specific questions about recent notes. Try measuring retrieval quality with and without the planning call before assuming it is necessary.

Fix 4: Leave the vector index alone

The index took 4.5 milliseconds. It is not the problem. Switching to a different embedding model, changing index parameters, or moving from an in-process index to a dedicated vector database will not meaningfully change the user-visible latency unless you first address the LLM call costs above.

The companion post on when grep outperforms an embedding-based index covers the retrieval decision itself: Don't Build a RAG Over Your Vault. Grep It.⁷

The honest caveat: speed is not the only problem

Fixing the latency does not solve everything. The pipeline measured above had a second finding that the timings table does not capture: "the top search hit was wrong more than half the time."¹ A pipeline that answers in three seconds with the wrong context note is not better than one that takes fourteen seconds with the right note. Speed and relevance are independent problems.

The source concentration in this post should be named directly. The timing figures come from a single practitioner's measurement on a single machine — a small 3B model on a mid-range laptop, one run. Arinay's data is well-documented and the analysis is careful, but the numbers are a measured example, not a controlled benchmark. The generalisable claim is the structure: Ollama's defaults (five-minute keep-alive, one concurrent request) explain why cold-start latency is common, and two serial LLM calls explain why the total wait accumulates. Those defaults are documented and verifiable; the absolute timings are illustrative.^{1 2}

Nielsen's 1993 response-time research established a rough threshold: at ten seconds, users lose the sense of a direct interaction and begin to feel they are waiting rather than working.⁸ A pipeline that reliably stays under ten seconds — warm model, single LLM call where possible, streaming answer — lands on the right side of that threshold for most users. A pipeline that only achieves that on the second question, after a cold start has already cost six seconds, does not.

Frequently Asked Questions

Why is my first question to Ollama slow?

The first question after a gap pays the cold-load cost — Ollama loads the model from disk before it can answer. On a mid-range laptop with a small 3B model, this loading step alone took roughly 6 seconds in one measured run. Subsequent questions in the same session skip that step and answer in a fraction of the time. Set keep_alive to a longer value to keep the model resident between questions.^{1 2}

Why does Ollama take so long to load a model?

Loading is bounded by disk I/O, not compute. The model's weights — which can be several gigabytes even for a small 3B parameter model — must be read from disk into memory before inference can begin. Research on serverless LLM infrastructure found that disk loading dominates cold-start time at every scale, from consumer laptops to multi-GPU servers.⁶ An SSD helps; eliminating the cold start by keeping the model resident helps more.

How do I keep an Ollama model loaded?

Pass "keep_alive": "30m" (or another duration, or "-1" for indefinite) in the request body of any Ollama API call. This overrides the five-minute default and sets how long Ollama waits before evicting the model. For a personal note-query pipeline, setting the value to match your session length removes cold starts entirely after the first query.^{2 3}

Does a faster vector database help with RAG latency?

Not meaningfully, if the timings above are representative. Vector search took 4.5 milliseconds of a 14-second total — roughly a third of one percent of the wait. A database ten times faster would save 4 milliseconds, which rounds to zero next to two serial LLM calls. The bottleneck is the language model calls, not the index.¹

Why is my local AI slow even with a good GPU?

If the pipeline makes two sequential LLM calls and the model reloads from disk between sessions, the GPU speed is not the constraint — the serial structure and the cold load are. Measure each stage individually before assuming hardware is the bottleneck. As Arinay found: "Time every stage of an AI pipeline separately before optimising any of it."¹

The search was never the problem. It never is. Build the pipeline, measure it in stages, and you will almost always find the same answer: two language model calls, cold, in series.

If your note app keeps files on your own device when you use a local model — and routes to your chosen provider when you use a hosted one — MNMNOTE is built around that distinction. mnmnote.com

References