The Tables in Your Notes Are the First Thing an AI Gets Wrong

A Markdown table is two-dimensional — rows crossed with columns. An AI reads your notes as a one-dimensional stream of text, split into chunks. Flatten the table, cut it at the wrong spot, and the second chunk keeps the rows but loses the header that told you what they meant.

That is the whole failure, and it is a failure of shape, not of intelligence. The engineers who build retrieval systems name it plainly. Matt Kauffman, a principal engineer at Ragie, describes what happens when a document is split by size: "A chunk may end in the middle of a column such that the subsequent chunk includes some of the table data, but without the table headers so contextual information is lost."¹ The header row is not decoration. In the GitHub Flavored Markdown spec, a table is defined as "an arrangement of data with rows and columns, consisting of a single header row, a delimiter row separating the header from the data, and zero or more data rows."² The header is a separate structural row — exactly the row a blind cut can strand. This post is vault hygiene, not a verdict on AI: structure-aware retrievers exist and are improving.³ The point is that your tables can be made safe today.

Why does an AI get the tables in your notes wrong?

Not because it can't read tables. Because default chunking flattens a table into a line of tokens, then splits that line at a fixed size — often mid-table. The chunk after the split inherits data rows with no header row above them, so a cell like $10M | 15% arrives with nothing to say which column is which.

There are two clean ways this breaks, and both are documented. The first is header loss: a chunk ends mid-column, and the next chunk carries data without its headers.¹ The second is the split record: "A chunk may end in the middle of a row so a record gets split across multiple chunks."¹ One row, two homes, neither complete.

The research literature calls the root cause by name. The TableRAG team — Xiaohan Yu, Pu Jian, and Chong Chen, in a paper accepted to EMNLP 2025 — write that "the prevailing practice of flattening tables and chunking strategies disrupts the intrinsic tabular structure, leads to information loss, and undermines the reasoning capabilities of LLMs in multi-hop, global queries."³ The villain is the shape — a grid pressed into a line — not the reader.

flowchart TD
  A[2-D table:<br/>header + data rows] --> B[Flattened to a<br/>1-D token stream]
  B --> C[Split at a<br/>chunk boundary]
  C -->|Chunk 1| D[Header row +<br/>its top rows]
  C -->|Chunk 2| E[Tail rows,<br/>no header row]
  E --> F[Rows lose<br/>their meaning]

Figure: How a table loses its meaning. A two-dimensional table (a header row plus data rows) is flattened into a one-dimensional token stream, then split at a fixed chunk boundary. The first chunk keeps the header and its top rows; the second chunk inherits the tail rows with no header above them, so those rows can no longer be interpreted.

The five-minute fix: keep tables small, repeat the header, isolate the table

Three moves, in effort order. Keep each table small enough to fit inside one chunk. When a table is too long to fit, repeat the header row over each block of rows so no split can strand them. And isolate a dense table in its own note, so it is chunked on its own terms.

This is not folk wisdom. It is the exact discipline the Ragie approach encodes. Kauffman's method: "If the chunk size accommodates the full table in markdown format, it is returned in 1 chunk" — otherwise the table is processed row by row, creating a new sub-table for as many rows as fit.¹ The goal state he names is the one you want in your own notes: table data that is "never disassociated from its table headers."¹

The five steps:

1. Count the rows. A table under roughly a dozen short rows usually fits inside a single chunk. Leave it alone.
2. Break long tables into blocks. Every ~10–15 rows, start a fresh table and repeat the header row. A split can no longer land in a headerless zone.
3. Repeat the header, not the whole table. You only need the one header row above each block — not a duplicate of the entire table.
4. Isolate a dense table. Move a big budget or comparison table into its own note so it is retrieved as a unit.
5. Add a plain-sentence caption. One sentence above the table naming what it holds ("Q3 spend by category, in USD") survives any split and re-anchors the numbers.

Here is the fix on a real table. A year of spend under a single header is one blind cut away from losing its columns:

| Month | Category | Spend (USD) | Notes          |
|-------|----------|-------------|----------------|
| Jan   | Cloud    | 1,240       | baseline       |
| Jan   | Tools    | 380         | annual renewal |
| ... 30 more rows ...                            |
| Dec   | Cloud    | 1,510       | scaled up      |

If a chunk boundary lands after June, every row from July onward arrives as Dec | Cloud | 1,510 | scaled up — with no header above it. The AI cannot tell whether 1,510 is dollars, a month, or a row number. The value is there; its meaning is in another chunk.

Break it into header-repeated blocks and every chunk stays whole:

### Spend — first half 2026
| Month | Category | Spend (USD) | Notes    |
|-------|----------|-------------|----------|
| Jan   | Cloud    | 1,240       | baseline |
| ...through June...                       |

### Spend — second half 2026
| Month | Category | Spend (USD) | Notes  |
|-------|----------|-------------|--------|
| Jul   | Cloud    | 1,290       | steady |
| ...through December...                 |

Now a split between the two blocks is harmless: each half is a complete table under its own header, and a heading tells the retriever where one unit ends and the next begins. The whole move took under five minutes and changed nothing about the numbers — only their shape.

The thirty-minute version: restructure a table-heavy note

When a single note holds several tables — a budget, a comparison, a log — spend half an hour giving each one room. Split the note by topic, one table per section under its own heading, add a plain-sentence summary above each table, and trim columns you never query. The structure you add is the structure the AI retrieves.

Work in this order. First, inventory: scroll the note and mark every table. Second, section it: give each table its own H2 or H3 heading, so the note is chunked at meaning boundaries rather than byte boundaries — the same reason how you split a note is its AI retrieval unit for prose, with the twist that a table is the one substructure headings alone can't protect. Third, summarize: write one sentence above each table restating its point in prose — that sentence is extractable even if the grid below it fractures.

Fourth, trim. A table with fourteen columns where you only ever ask about three is fourteen columns of split risk. Cut the columns you never query into a separate reference table, or drop them. Fifth, re-read the widest table aloud, row by row, header-first — if a single row makes no sense without scrolling up to the header, a chunk boundary will make no sense of it either.

Structure-preserving methods report real gains from exactly this shape. The Structure-Aware Chunking preprint (Guttal et al., 2026, a work in progress on arXiv) reports that its method "reduces chunk count by up to 40% and 56% compared to standard recursive and key-value based baselines," and lifts retrieval quality — "improves MRR from 0.3576 to 0.5945 in a hybrid setting and increases Recall@1 from 0.366 to 0.754 in BM25-only retrieval" — on its test set.⁴ Treat those as a research signal, not a promise: the direction is what matters, and the direction is that keeping a table intact helps.

Common mistakes

The failures cluster around shape, not content. A giant table that no chunk can hold. A header written once, far above two hundred rows. Merged cells and multi-row headers that flatten into nonsense. And the belief that the AI simply reads tables the way you do — it does not; it reads whatever the chunk boundary left it.

• The wall-of-rows table. Two hundred rows under one header is two hundred rows that any split can orphan. Break it into header-repeated blocks.
• The lonely header. A header written once, at the very top, protects only the first chunk. Repeat it.
• Fancy structure that flattens badly. Merged cells, multi-row headers, and nested tables linearize into ambiguous token soup. Prefer a plain single-header-row table.
• Format assumptions. How a table is fed to a model changes the answer. The "Table Meets LLM" study (Sui et al., WSDM 2024) found "performance varied with different input choices, including table input format, content order, role prompting, and partition marks."⁵ Their honest framing: "there is a lack of comprehensive studies on whether LLMs genuinely comprehend this data."⁵
• Assuming the tool will fix it. Some retrievers are table-aware and will. Many default pipelines are not. Shape your notes so the answer does not depend on which one you happen to be using.

How this works in a plain-Markdown vault

All of this is easier when your notes are plain Markdown files you can open, read, and reshape yourself. A table in a .md file is text you can see — you can shorten it, repeat its header, or move it into its own note, and the file on disk stays yours to re-chunk however a tool asks.

That is the quiet advantage of local-first, plain-text notes: the table is not locked inside a proprietary block you cannot inspect. It is Markdown, offline, on your own device — a grid you can diff, restructure, and hand to any AI on your terms. The hygiene above is not a workaround for a closed format; it is ordinary editing of a file you already control.

Frequently Asked Questions

Short answers to the questions people actually type when the AI misreads a table in their notes. Each one traces back to the same cause — a two-dimensional grid flattened to a line and cut at the wrong place — and each fix is one you can apply to the note yourself in minutes.

Why does the AI get the numbers in my table wrong? Usually because a chunk boundary separated the numbers from their header row. A retrieved cell like "$10M | 15%" is meaningless once the header naming those columns is in a different chunk — the value survives, its label does not.¹ Keep the table small enough to stay whole, or repeat the header so no split can strand the numbers.

Should I repeat the header row when I split a table? Yes — that is the single most effective fix for a long table. If a table is too big to fit in one chunk, break it into blocks and repeat the header above each block, so table data is "never disassociated from its table headers."¹ A split that lands between blocks then still leaves every row with a header above it.

How do I keep a table together in one chunk? Make it small enough to fit, or give it its own note. Kauffman's rule is that a table which fits the chunk size "is returned in 1 chunk"; otherwise it is broken up row by row.¹ Trimming unused columns and isolating a dense table in a dedicated file are the two reliable ways to keep a whole table in one retrievable unit.

Are Markdown tables good for AI and RAG? A plain single-header-row Markdown table is one of the friendlier structures to retrieve — but how it is chunked and fed to the model matters more than the format label. The "Table Meets LLM" study found model performance "varied with different input choices, including table input format, content order, role prompting, and partition marks."⁵ Shape beats format.

Why does my AI answer with the wrong row or mix up columns? Because flattening breaks the grid. Yu, Jian, and Chen note that flattening and chunking "disrupts the intrinsic tabular structure, leads to information loss, and undermines the reasoning capabilities of LLMs in multi-hop, global queries."³ When rows fragment across chunks, the model stitches together cells that were never adjacent — a wrong-row or mixed-column answer.

Does this mean an AI can't read tables? No. This is vault hygiene, not a capability verdict. Naive, default chunking mangles tables — but structure-aware retrievers and chunkers exist and are improving, which is exactly what systems like TableRAG and Structure-Aware Chunking are for.³⁴ The fixes here are the parts you control; the tooling is catching up on its side.

How do I chunk tables for retrieval? Keep the full table in one chunk when it fits; when it does not, split by row and carry the header into each piece.¹ In your own notes the equivalent is: small tables, repeated headers on long ones, dense tables isolated in their own files, and a plain-sentence summary above each. Chunk at meaning, not at byte counts.

Your notes are only as legible to an AI as their worst-shaped table — and the shape is the one thing you can fix before you ever ask a question.

For notes that stay plain Markdown on your own device, where every table is text you can reshape yourself, there is mnmnote.com.