The Chat Runs on Your Machine — the Embedding Step May Have Uploaded Every Note

Running a local LLM does not make your notes private by itself. Building the retrieval index — the step that lets the model search your notes — is a separate operation, and by default it calls OpenAI's API with every chunk of text you feed it. The chat model runs on your machine. The embedding step may have already uploaded your library.¹

That gap is not a secret. LlamaIndex, one of the most-used frameworks for building retrieval-augmented chat pipelines, states it plainly in its own privacy documentation.² But that page lives two clicks from the quickstart that most people run without reading it. The quickstart produces a working pipeline — one that silently touches OpenAI every time you index a new note, without any warning in the terminal.

This post names the mechanism, gives you two ways to verify whether your pipeline is outbound, shows the configuration change that makes the embedding step local, and states the retrieval-quality trade-off honestly. The chat path — whether your text-generation step is private — is addressed separately in Run Private AI on Your Own Notes (No Cloud).³ This piece is about the indexing path: what happens before you ask your first question.

What "local" covers — and what it does not

Running a chat model on your own hardware makes the text-generation step private — the model receives a prompt, predicts tokens, and returns a response without calling any external service. That is the guarantee tools like LM Studio and Ollama provide, and it holds for generation specifically.

A retrieval-augmented generation pipeline is not a single step, though. Before the model can answer a question about your notes, the pipeline must: load your documents, split them into chunks, convert each chunk into a numeric vector, store those vectors in an index, and at query time retrieve the closest-matching chunks before passing them to the model. The local LLM handles only the last of those operations — generation. The earlier steps are a separate system, running a separate model, with its own API calls.

That separate model is the embedding model. It handles the conversion of text into vectors. And by default, it calls OpenAI.

The embedding step is where notes leave your machine

Embedding converts your note text into dense numeric vectors — a representation the retrieval layer uses to find semantically similar chunks through mathematical comparison. Those vectors get stored in the index; your original text has to travel through an embedding model to produce them. That travel is the exposure point.

By default, LlamaIndex uses text-embedding-ada-002 from OpenAI.¹ When you build or rebuild a vector index over your notes, LlamaIndex sends each document chunk to the OpenAI embeddings API and receives a vector in return. OpenAI prices this model at $0.10 per million tokens.⁴ The vendor states the consequence directly: "By default, LLamaIndex sends your data to OpenAI for generating embeddings and natural language responses."²

The embedding model is pluggable — LlamaIndex is explicit that you can swap it — but pluggable means the default runs unless you replace it.¹ A related post on shared agent memory describes the plug-in architecture of the embeddings layer in more depth.⁵ The point here is that the replacement requires an explicit configuration change. There is no flag to set for privacy; there is a different component to install.

LlamaIndex sends embedding requests to OpenAI in batches of 10 by default.¹ For a library of several hundred notes, that is dozens to hundreds of API calls at index-build time — each carrying the text of your notes.

How to confirm whether your pipeline is outbound

Two checks reveal the same fact from different angles. The first observes outbound connections in real time during an active re-index. The second is a binary test — local or not — applied to the embedding model loader itself, confirming whether the model was already cached on your machine or required a network fetch.

During a re-index, run this in a separate terminal:

lsof -i -P | grep api.openai.com

The -i flag lists all open Internet connections on the machine.⁶ Any output while the pipeline is building an index confirms the embedding step is reaching OpenAI. The command works on macOS and Linux. Run it while the index build is active — the call happens at index time, not at query time.

To test whether the embedding model is cached locally:

HF_HUB_OFFLINE=1 python your_index_script.py

HF_HUB_OFFLINE is the primary env var for offline mode in the HuggingFace Hub library. Its documented behavior: "If set, no HTTP calls will be made to the Hugging Face Hub. If you try to download files, only the cached files will be accessed. If no cache file is detected, an error is raised."⁷ Set it before running any script that loads a HuggingFaceEmbedding model. Success means the model is cached and runs locally; an offline error means the model was never downloaded — a prior run made the network request to fetch it.

HF_HUB_OFFLINE tests HuggingFace Hub calls specifically. The OpenAI embedding endpoint is a different network destination — the lsof check handles that.

How to stay local at every step

LlamaIndex supports HuggingFace embedding models that run on-device once downloaded. Swapping the default requires one new package and one configuration line. After the initial download, no outbound network call is made at index-build time — the model runs from your local cache.

# pip install llama-index-embeddings-huggingface
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core import Settings

Settings.embed_model = HuggingFaceEmbedding(
    model_name="BAAI/bge-small-en-v1.5"
)

LlamaIndex documents this directly: "To save costs, you may want to use a local model," and provides this exact configuration.¹ The BAAI/bge-small-en-v1.5 model downloads once to your HuggingFace cache — approximately 30 MB on disk — then runs offline for every subsequent index build. Add HF_HUB_OFFLINE=1 to your run environment to guarantee no further Hub traffic.

Confirm the swap with lsof -i -P | grep api.openai.com during a fresh re-index. No output means the embedding step is now local.

One practical note: the package name above (llama-index-embeddings-huggingface) reflects the modular structure LlamaIndex adopted in its 0.10.x release. If the name changes in a future version, the embeddings documentation is the authoritative source.

What you give up — the honest trade-off

Local embedding models are smaller than OpenAI's. Smaller means faster, free of API costs, and fully private — but also means the model maps text to semantic space with less precision. That precision difference affects retrieval quality, which determines how relevant the note chunks the model receives actually are.

text-embedding-ada-002 was trained on a large, diverse corpus and captures nuanced semantic relationships well. BAAI/bge-small-en-v1.5 is capable for its size class — it performs competitively on public retrieval benchmarks among small, open models — but it is not equivalent to the OpenAI default. On a general personal-note corpus written in everyday language, the practical retrieval difference is often modest. On specialized vocabularies — medical notes, legal documents, dense technical research — the gap can be noticeable enough to affect which notes surface for a given question.

The embedding model you choose also locks in your existing index. Queries must use the same model that built the index — switching models means rebuilding every vector from scratch. Choosing which model to use, and the full re-embedding cost of switching later, is addressed in this post on index model lock-in.⁸

Test both models on a sample of your actual notes before committing. Private retrieval that is good enough for your use case beats the highest benchmark score at the cost of sending your library to a cloud API.

The scope of the risk, precisely stated

The outbound embedding call happens once — at index-build time — not on every chat query. At query time, the pipeline encodes your question into a vector, searches the locally stored index for similar vectors, and passes the matching chunks to the local LLM. That query-time flow can be entirely local. The exposure is at construction time, not at every use.

When you rebuild your index, every document chunk in scope is sent to the embedding API. The notes you had at initial setup were sent when you first built the index. New notes are sent on the next re-index. The frequency of exposure is controlled by how often you run index builds — not by how often you chat. That makes the risk bounded and batch-shaped: a defined event with a start and an end, not continuous background monitoring.

Where the index artifact lives after that build — in a local file or a hosted vector store — is a separate custody question addressed in the post on index custody.⁹ LlamaIndex adds a relevant caution: "each vector store has its own privacy policies and practices, and LLamaIndex does not assume responsibility for how it handles or uses your data."² The embedding step and the index-storage step are two distinct places where data handling can diverge from what you assume. A pipeline that embeds locally but stores the index in a cloud vector database has swapped one exposure for another.

Frequently Asked Questions

Does LlamaIndex send my notes to OpenAI?

Yes, by default — at index-build time. LlamaIndex uses text-embedding-ada-002 from OpenAI as its default embedding model. Every document chunk is sent to the OpenAI embeddings API when you build or rebuild your vector index. This does not happen during chat queries — only when the index is being constructed or updated.¹

How do I use local embeddings with LlamaIndex?

Install llama-index-embeddings-huggingface and set Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5") before building your index. The model downloads once to your local cache. On subsequent runs, set HF_HUB_OFFLINE=1 to confirm no Hub requests are made. Verify with lsof -i -nP | grep api.openai.com during a re-index.¹

Is LlamaIndex private?

Only if you explicitly configure both the LLM and the embedding model to be local. The default setup sends your document text to OpenAI at index-build time. A fully private pipeline requires a local embedding model and a vector store that keeps the index on your machine.²

How do I check if my RAG stack sends data to OpenAI?

Run lsof -i -P | grep api.openai.com in a terminal while your pipeline is actively building an index. Any output confirms the embedding step is reaching OpenAI. Run it during the index build — not during a chat query, since query time does not trigger embedding calls.⁶

I run the chat model locally. Why is data still leaving my machine?

Running the chat (generation) model locally makes the text-generation step private. It does not affect the embedding step, which is a separate operation that runs before any chat occurs. The local LLM and the embedding model are independent components with separate configurations — changing one does not change the other. See the companion post on the chat path for how generation works.³

What does HF_HUB_OFFLINE=1 tell me about LlamaIndex embeddings?

It forces the HuggingFace Hub client to use only locally cached files and make no network calls. If you set it and run a script loading a HuggingFace embedding model — success means the model is cached and running locally; an error means it was never downloaded and a prior run made a network request to fetch it. That error is useful information, not a failure.⁷

The privacy of a local-LLM setup is only as good as its least-local step. The embedding call is an engineering default, not an architectural requirement — and one configuration change removes it.

MNMNOTE keeps your notes on your own device, in open Markdown, with no account required. If you are thinking about where your notes live and who can reach them, mnmnote.com is built around that question.