How You Split a Note Decides What an AI Finds in It

When an AI answers a question from your notes, it does not read the whole file. It retrieves a slice — a chunk — and answers from that. The chunk is decided by how you wrote the document. So a clean heading hierarchy and one idea per section are not formatting habits. They are a retrieval strategy you already own.

This is the part most people skip when they build a personal AI over their own notes. They tune the database, swap the embedding model, raise the number of results, and still get vague answers. The lever they never touch is the one they control completely: the shape of the input. A wall-of-text note splits into useless fragments no matter how good the rest of the pipeline is. A note with clean ## sections splits into clean answers.

What is a chunk, and why does it decide your answer?

A chunk is the unit of text an AI retrieval system stores and returns. Your note is cut into pieces, each piece is indexed, and a question retrieves only the few matching pieces — those go to the model. The model answers from the chunk, not the file. Split one idea across two chunks and neither is complete.

This is the architecture behind retrieval-augmented generation, or RAG — the standard way a local AI reads a body of documents it was never trained on. The model never sees your whole vault. It sees a handful of retrieved fragments. Everything downstream of retrieval is only as good as the fragment that came back.

The size and boundary of that fragment are set by the chunking strategy — the rule that decides where one piece ends and the next begins. Chunking is not an exotic setting buried in a config file. For a markdown note, the most reliable boundary is the one you already wrote — the heading.

Why does chunking matter more than the database?

Chunking matters more than the database because the database can only rank what the chunker handed it. In a controlled evaluation, the research company Chroma found that "the choice of chunking strategy can have a significant impact on retrieval performance, with some strategies outperforming others by up to 9% in recall." ¹ That gap is set upstream — by the split, not by the store.

The point is easy to miss because the database is the visible, brand-name part. The chunker is invisible. So when answers go wrong, people upgrade the store. The Firecrawl engineering team, writing about chunking for RAG in 2026, puts the order of operations plainly: "This is why chunking strategy matters more than vector database choice for many applications." ² And the sharper version: "Poor chunking degrades retrieval even on the fastest vector database." ³

A fast database that retrieves a fragment cut mid-thought returns that broken fragment quickly. Speed does not repair a bad boundary. The boundary is upstream, and upstream is where you write.

Two honest bounds before you read further. That 9% is a measured range across strategies in Chroma's own evaluation, on their corpus — an upper figure, not a universal law. And recall gains of this kind are corpus- and embedding-model-dependent, as the next section shows. Structure is the lever you control. It is not a guarantee.

Does the structure of a markdown note actually help retrieval?

Yes — and the cleanest reason is structural, not statistical. A markdown file is already organized into sections by its headers, so the section is a natural retrieval unit. The LangChain documentation states the rationale directly: "a markdown file is organized by headers. Creating chunks within specific header groups is an intuitive idea." ⁴

This is the whole move. Instead of cutting the text every N characters and hoping the cut lands somewhere sensible, you cut on the headers you already wrote. Each ## becomes a chunk boundary. The Weaviate engineering team describes the gain from respecting that structure: such a split "respects the natural organization of the text, rather than splitting it randomly." ⁵ Random splits land mid-sentence. Header splits land where your thought already ended.

The benefit shows up in measurement, with a caveat attached. In Snowflake's finance-RAG evaluation, markdown-aware chunking that splits on section headers boosted accuracy by roughly 5–10% over fixed-size splits — though that advantage lessens when the model can already see the whole document. ⁶ Structure helps most exactly when the model cannot hold everything at once, which is the normal case for a large note collection.

One worked example: the same note, two ways

Here is the difference a heading makes, shown as the two chunks a splitter would produce from the same content. The first note has no internal structure, so a fixed-size splitter cuts it wherever the character count runs out — often mid-idea:

# Project Atlas notes

We kicked off Atlas in March. The goal is to cut onboarding time
from three weeks to one. Budget is approved through Q3. The main
risk is the data-migration step, which depends on the legacy export
finishing first; if that slips, everything slips. Sarah owns
migration. We decided to ship the new flow behind a flag and roll
it out to 10% of accounts before [CHUNK CUTS HERE] going wider.
The rollback plan is to flip the flag off — no data is destroyed,
so reverting is safe. Next review is the second Friday of the month.

Ask "what is the rollback plan?" and the splitter may hand the model the first chunk, which ends before the rollback sentence. The answer is absent from the retrieved slice. Now the same content, written with one idea per section:

# Project Atlas notes

## Goal
Cut onboarding time from three weeks to one. Budget approved through Q3.

## Risks
The main risk is the data-migration step; it depends on the legacy
export finishing first. If that slips, everything slips. Sarah owns
migration.

## Rollout
Ship the new flow behind a flag. Roll out to 10% of accounts before
going wider.

## Rollback plan
Flip the flag off. No data is destroyed, so reverting is safe.

## Next review
Second Friday of the month.

A header-aware splitter cuts on each ##, so "Rollback plan" is its own complete chunk. Ask the rollback question and the system retrieves a slice that contains the entire answer — title and all. You did not configure anything. You wrote a heading.

The method: structure a note so the natural chunk is the right chunk

The method is four habits, in order. Each one makes the section a self-contained unit that survives being retrieved alone. None of them requires a tool you do not already have — a header-aware splitter does the cutting; your job is to give it boundaries worth cutting on.

1. Use a clean heading hierarchy. Give every distinct topic its own ##, and use ### for sub-points beneath it. The headers are the chunk boundaries, so a heading that means something produces a chunk that means something.
2. Keep one idea per section. A section should answer one question. When a chunk is retrieved on its own, everything needed to understand it should be inside it — no "as mentioned above," no thread continued three headings later.
3. Keep code, tables, and lists intact under one heading. These break worst when split. A table cut in half is two meaningless tables. Put the whole artifact under one section so it travels as one chunk.
4. Write a heading that states the answer, not just the topic. "Rollback plan" beats "Notes." "How sync handles conflicts" beats "Sync." The heading is often retrieved with the chunk, so a descriptive heading is free context for the model.

Why one idea per section is the load-bearing habit

The reason is that retrieval returns sections in isolation, so a split idea arrives broken. If the setup is under one heading and the conclusion is under another, a query that matches the conclusion retrieves a chunk with no setup, and a query that matches the setup retrieves a chunk with no conclusion. Neither answers the question. Keeping the idea whole under one heading keeps the chunk whole.

A clinical-decision-support study published in Bioengineering in November 2025 measured this effect directly. Comparing chunking methods for a medical RAG system, the authors found that "adaptive chunking achieved the highest accuracy—87% (Likert 2.37 ± 0.72) versus baseline 50% (1.63 ± 0.72; p = 0.001)." ⁷ The structure-aware method nearly doubled accuracy over the naive baseline.

Read that number carefully. It is clinical-domain research — not medical advice, and not a substitute for professional care — and the authors are explicit that it does not transfer automatically: "The generalizability of this approach beyond postoperative rhinoplasty remains to be established… specialty-specific terminology, documentation style, and evidence density can influence performance." ⁸ The 87% is corpus-specific. What generalizes is the mechanism the study names: structure-aware splitting beats splitting by raw token count, because, as the authors put it, advanced methods work by "organizing text around meaningful units rather than token counts, preserving context and improving retrieval accuracy." ⁹ A heading is the simplest meaningful unit there is.

What chunk size should you use?

There is no universal best chunk size, and any guide that names one is overselling. A 2025 multi-dataset analysis by Bhat and colleagues found the optimum shifts with the data: small chunks suit concise, fact-based answers, larger chunks suit broader context, and the right size depends on the embedding model too. ^{10 11} Size is a moving target.

In their words, "smaller chunks (64-128 tokens) are optimal for datasets with concise, fact-based answers, whereas larger chunks (512-1024 tokens) improve retrieval in datasets requiring broader contextual understanding." ¹⁰ The same paper found embedding models "exhibit distinct chunking sensitivities." ¹¹

This is freeing, not discouraging. If size were a fixed dial, you would have to tune it. Because it is corpus-dependent, the better default is to let meaning set the boundary — split on the heading, and let each idea run as long or short as it needs to be. A header-aware split adapts to your content on its own — which is exactly what the size research says you want.

For a starting baseline when you do need a number, a plain recursive splitter is already strong. Firecrawl reports that a RecursiveCharacterTextSplitter in the 400–512 token range "delivered 85-90% recall in Chroma's tests without the computational overhead, making it a solid default for most teams." ¹² It is "the default choice for 80% of RAG applications because it balances simplicity with structure." ¹³ You do not need exotic tooling. You need structured input.

Common mistakes that wreck retrieval

These three habits quietly destroy chunk quality, and all three sit upstream of any database setting — which is why no amount of downstream tuning repairs them. They are the failure modes the method above is built to prevent, and each one is the inverse of one of the four habits. Recognize the shape, then fix the input.

• The wall-of-text note. One heading, two thousand words, no internal structure. A fixed-size splitter cuts it at arbitrary character counts, so every chunk straddles a boundary the writer never intended. The fix is the cheapest one in this post: add headings.
• Headings that do not mean anything. "Misc," "Notes," "Stuff," "More." A generic heading produces a generic chunk that matches no specific query well. Name the section after the answer it contains.
• Splitting one thought across two sections. Setup under "Background," conclusion under "Decision," three headings apart. Retrieval pulls them separately and the model sees half an argument. Keep the complete thought under one heading. As Firecrawl puts it, the goal is that "headers stay with their content. Sections break at natural boundaries instead of mid-sentence." ¹⁴

How this fits a note you actually own

A note kept in plain Markdown, on your own device, is already in the format a header-aware splitter wants. The natural chunk — a ## section — is the right chunk, with nothing to convert in between. The document you wrote is the retrieval unit, and you control both ends: how it is written, and how it is read.

That is the quiet advantage of writing in open Markdown rather than a proprietary format. The structure that helps a human navigate a long note is the same structure that helps an AI retrieve the right part of it. You build the index once, by writing well, and both readers — human and machine — benefit from the same headings. For the navigation side of that habit, see Navigate a Long Note by Its Outline; for the storage side, Own the Vector Index Next to Your Files.

Frequently asked questions

How does chunking affect RAG quality? Chunking sets the boundary of every fragment the AI retrieves, so it shapes whether the retrieved slice actually contains the answer. In Chroma's evaluation, strategies differed by up to 9% in recall. ¹ A clean split returns a complete idea; a bad split returns half of one. The database can only rank what the chunker handed it.

What is the best way to structure notes for AI retrieval? Split on headings. Give each topic its own ##, keep one idea per section, and write headings that state the answer. LangChain's documentation calls chunking by header group "an intuitive idea" because a markdown file is already organized by headers. ⁴ The section you wrote becomes the chunk the AI retrieves.

Why does my RAG give vague or wrong answers? Usually the chunking, not the database. If a note is a wall of text, it splits into fragments that straddle ideas, and the model answers from a broken slice. Firecrawl notes that "poor chunking degrades retrieval even on the fastest vector database." ³ Add headings and keep one idea per section before you change anything downstream.

Do I need a better vector database to fix retrieval? Often not. Chunking strategy "matters more than vector database choice for many applications," and the input you control sits upstream of the store. ² Fix the document structure first; a faster database only returns a bad fragment faster.

What chunk size should I use? There is no universal best. Research by Bhat and colleagues found smaller chunks (64–128 tokens) suit concise, fact-based answers while larger chunks (512–1024 tokens) suit broader context, and that the right size depends on the embedding model too. ¹⁰ Splitting on headings sidesteps the question by letting meaning set the boundary; a recursive splitter at 400–512 tokens is a strong default if you need a number. ¹²

Does markdown structure actually help retrieval? Yes, with a caveat. Snowflake's finance-RAG evaluation found markdown-aware, header-based chunking boosted accuracy by roughly 5–10% over fixed splits, though the gain shrinks when the model can see the whole document. ⁶ Structure helps most when the collection is too large to fit in the model's working memory at once — see The Context Window Is Working Memory.

The cleanest way to improve what an AI finds in your notes is to write notes worth finding things in. Headings are the strategy; you have had them all along.

For a personal AI you can actually audit, mnmnote.com keeps your notes in open Markdown on your own device, where the section you wrote is the slice an AI retrieves.