The Answer Is in Two Notes — and Your AI Only Reads One

Your vault holds both halves of the answer, and the assistant reads one. A question that spans two notes needs two retrievals, and the second is invisible until the first resolves — so a single search pass fetches half the evidence. Ask hop by hop, then write one note that restates both facts.

The failure has a name and a measurement. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith and Mike Lewis called it the compositionality gap — "We measure how often models can correctly answer all sub-problems but not generate the overall solution, a ratio we call the compositionality gap."¹

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot and Ashish Sabharwal named the other half — the retrieval half — reporting that "we observe that this one-step retrieve-and-read approach is insufficient for multi-step QA."²

Both teams measured public documents — trivia about celebrities, Wikipedia articles, news wire copy — on the models of 2022 to 2024. Nobody has benchmarked a personal note vault. What transfers is not a percentage but a mechanism, and the mechanism is corpus-independent: the second thing you need to look up is defined by the answer to the first.

Your notes have that shape all the time.

Why does my AI miss answers that are in my notes?

Because the retriever scores your notes against the question you typed, and your question names only the first fact. The second note is relevant to the answer, not to the question, so nothing pulls it into the context window. The model then writes a fluent, confident reply from half the evidence.

Picture the concrete case. In March you wrote a decision note: the billing service will run on Postgres, not DynamoDB. In June you wrote a retro that never repeats the decision and only records the reason — transactional writes across three tables, and read volume that never exceeded forty queries per second. Two files, twelve weeks apart, no shared phrasing beyond the word billing.

Now ask the composed question — "why did I choose Postgres for billing?" The retriever embeds that sentence, finds the March note, and stops. The June note does not mention Postgres. It is the answer to hop two, and hop two was never asked.

Trivedi and colleagues put the dependency in one line: "Here, what to retrieve depends on what has already been derived, which in turn may depend on what was previously retrieved."² That is the whole failure. A one-shot search cannot know the second hop exists until the first hop resolves.

Name the failure precisely before you fix it. Four different things go wrong in a chat over your notes, and they have four different remedies:

• Nothing relevant was retrieved. The fact is absent, or the index cannot see the file. Teach the assistant to say so — Make Your AI Say 'It's Not in Your Notes'.
• Too much near-relevant text was retrieved. Distractor clutter degrades the answer, covered in Your Vault Has Too Many Almost-Right Notes.
• The question needed every note of a kind, not two specific ones. That is an enumeration problem — Your AI Can't Count Your Notes.
• Half the evidence arrived and the answer read fine anyway. That is this post.

The fourth is the hardest to notice, because nothing looks broken. Before blaming composition, confirm retrieval fires at all — plant a unique string and query for it, the method in Bury a Canary in Your Notes to Test Your RAG. A canary proves one chunk is reachable. It cannot detect a composition failure.

This is not a niche grievance. On Hacker News, "Vector databases are the wrong abstraction" drew 493 points in October 2024, "Show HN: FastGraphRAG — Better RAG using good old PageRank" drew 457 points in November 2024, and "Ask HN: How are you doing RAG locally?" drew 413 points in January 2026.³ The complaint underneath all three is the same — one similarity search returns a relevant note, not all the notes the question needs.

What is the compositionality gap?

The compositionality gap is Press and colleagues' name for how often a model answers every sub-question correctly and still gets the combined question wrong. Measured on the GPT-3 family in 2022 and 2023, it sat at roughly 40% — and it stayed there as the models got bigger.

Their finding, verbatim: "we find that the compositionality gap remains at a roughly constant 40% between different model sizes and training techniques, with no apparent improvement from scale."¹ On the same evaluation set, "GPT-3 (davinci-002) correctly answers 45.4% of the 2-hop questions."¹ Not hopeless. Not reliable either.

The scale result is the one worth sitting with — "as model size increases we show that the single-hop question answering performance improves faster than the multi-hop performance does, therefore the compositionality gap does not decrease."¹ Bigger models learned more facts without learning to join them.

Press and colleagues state the implication plainly: "This surprising result suggests that while more powerful models memorize and recall more factual knowledge, they show no corresponding improvement in their ability to perform this kind of compositional reasoning."¹

Waiting for a better model is not a plan.

The honest counterweight sits in the paper's own footnote. On the same dataset the authors report: "We observed that they exhibit a compositionality gap, estimated at 42.9% for ChatGPT and 23.0% for GPT-4."¹

GPT-4's gap was much smaller — and the authors immediately flag why the number may not mean what it looks like: "Since we publicly released CC before the launch of GPT-4, it may have been included in its training set, so that result may be invalid."¹ Quote one of those sentences and you owe the reader the other.

They fence their own ceiling too: "While we showed here that the compositionality gap stays at around 40% for models of size between 1 billion to 175 billion parameters, we do not have access to, and so did not experiment with, models that are larger than 175 billion parameters."¹

Every model in this post is a 2022-to-2024 model — so the durable claim is the direction, not the constant. Composition is a separate failure mode that scale did not close on its own.

Retrieval and composition are two different failures

One failure is that hop two never arrives. The other is that the model fumbles the combination even when both facts sit in front of it. Yixuan Tang and Yi Yang separated them cleanly: GPT-4 reached 0.89 accuracy on multi-hop questions when handed the correct evidence, and 0.56 when it had to retrieve that evidence itself.⁴

That pair of numbers is the architecture of the whole problem. Hand the model perfect evidence and it still misses roughly one question in nine. Make it find the evidence first and accuracy falls by a third. Fixing retrieval does not fix composition, and fixing composition does not fix retrieval.

The benchmark behind those numbers is a news corpus, not a vault. MultiHop-RAG covers "six different types of news articles, covering 609 distinct news, with an average of 2,046 tokens," and its 2,556 queries are weighted toward exactly the shape described above — 42.18% need two pieces of evidence, 30.48% need three, 15.56% need four.⁴

Tang and Yang's verdict, filed as an arXiv preprint in January 2024: "we find that existing RAG systems are inadequate in answering multi-hop queries, which require retrieving and reasoning over multiple pieces of supporting evidence."⁴ They ran two experiments — one on embedding models for evidence retrieval, one on reasoning over that evidence — and concluded of both: "Both experiments reveal that existing RAG methods perform unsatisfactorily in retrieving and answering multi-hop queries."⁴

Two experiments. Two failures. One symptom.

The retrieval half has a documented remedy, and its shape is the tell. Trivedi and colleagues built IRCoT, which interleaves a retrieval step with each step of the model's reasoning instead of retrieving once at the start.

It "is substantially more effective than the baseline, one-step, question-based retrieval by 11-21 recall points under a fixed-budget optimal recall setup," and "it also leads to substantial improvement (up to 15 F1 points) in downstream few-shot QA performance and reduces factual errors in generated CoT by up to 50%" — measured across HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC as of ACL 2023.²

You cannot install IRCoT in your notes app. But you can copy its idea by hand, and that costs five minutes.

The five-minute version: ask hop by hop

Stop asking the composed question. Ask the first hop, read the answer, then paste that answer into the second question and ask again. Press and colleagues' self-ask prompt does exactly this, and on their own evaluation set it scored 79.6% with InstructGPT-davinci-002 against 45.7% for generic step-by-step prompting.¹

Their comparison, stated in full: "it obtained 45.7%/1.1% accuracy with InstructGPT-Davinci-002/Davinci whereas self-ask obtains 79.6%/54.2% accuracy with those models on CC."¹ Those are 2022-to-2023 models on a public trivia set — read the direction, not the digits. Decomposing the question beat telling the model to think harder, by a margin no prompt-polish closes.

Do it by hand:

1. Write the composed question down and mark the noun you do not yet know. In "why did I choose Postgres for billing?", the unknown is the reason, and the reason lives somewhere you have not named.
2. Ask hop one alone. "What did I decide about the billing database, and when?"
3. Open the file it cites. Confirm the date and the wording yourself. A wrong hop one poisons everything downstream.
4. Rewrite hop two with hop one's answer substituted in literally. "In the June retro, what reasons did I record for the March billing database decision?"
5. Ask hop two.
6. Ask the composed question once more, pasting both retrieved passages in as context. Now the model is composing, not searching.
7. If a hop comes back empty, stop. The fact may genuinely not be in your vault, and a model asked to fill that hole will invent one.

Query decomposition is not a folk remedy — it is the shipped industry workaround. The LlamaIndex project documents a sub-question query engine that "first breaks down the complex query into sub questions for each relevant data source, then gather all the intermediate reponses and synthesizes a final response."⁵ The typo is theirs — the quote is verbatim. The research fixes the pipeline. You can only change the inputs, so change them the same way.

One scoping note, because it matters — every hop you ask sends the text you choose to the provider you choose. Asking three narrow questions instead of one broad one usually sends less of your vault, not more.

The thirty-minute version: write the synthesis note

Asking hop by hop fixes one question, once. To fix it permanently, write a note whose body restates both facts in plain sentences, so that a single retrieval brings back the whole answer. One note per composed question, dated, linking to both sources — and never a copy-paste fork of either.

The mechanism is dull — and that is the point. Retrieval scores chunks. If one chunk contains both facts and the composed claim, the composed question matches it directly, and there is no second hop to miss. You are not making the model smarter — you are removing the hop.

Here is the shape, copy-pasteable:

---
title: Why billing runs on Postgres
type: synthesis
question: "Why did I choose Postgres for the billing service?"
sources: ["2026-03-11-billing-db-decision", "2026-06-02-billing-retro"]
updated: 2026-07-28
---

## The composed answer

Billing runs on Postgres because the March 2026 design required transactional
writes across three tables, and the June 2026 retro confirmed sustained read
volume never passed 40 queries per second — so the scale argument for a
key-value store never applied.

## Fact one — the decision (2026-03-11)

Chose Postgres over DynamoDB for the billing service. Recorded in
`2026-03-11-billing-db-decision`. No reason written down at the time.

## Fact two — the reason (2026-06-02)

The retro recorded two constraints: three-table transactional writes, and
measured peak read volume of 40 qps. Recorded in `2026-06-02-billing-retro`.

## What would change this answer

Sustained reads above ~500 qps, or a schema that drops cross-table
transactions. Revisit then; until then this note is the answer.

Four rules make it work:

• Restate, don't link. A wiki-link is not retrievable text. This is the opposite job from the index note in You Don't Need a Graph. You Need One Index Note. — that artifact links out for a human to navigate; this one holds the facts in its own body for a machine to retrieve. Same shape on screen, opposite purpose.
• Put the composed answer in the first section. Chunkers split on structure, and the first chunk is the one that has to carry the whole claim on its own.
• Write it in the question's vocabulary. If you will one day ask "why Postgres for billing", those words belong in the body, not only in the source notes.
• Date it and name its expiry. A synthesis note that cannot be invalidated is a rumor with frontmatter.

Common mistakes

Most of the damage comes from five habits: duplicating instead of synthesizing, waiting for a smarter model, assuming a text search will save you, blaming composition before checking retrieval, and letting the note rot. Each one is cheap to avoid and expensive to leave in place.

• Copy-pasting both notes into one file. You have just built a near-duplicate of two notes you already have, and near-duplicates crowd the retrieved set with redundant text — the failure documented in Duplicate Notes Poison Your AI's Retrieval and Your Vault Has Too Many Almost-Right Notes. A synthesis note compresses two facts into new sentences — it should be shorter than either source, not longer than both.
• Waiting for the next model. Press and colleagues measured single-hop accuracy improving faster than multi-hop accuracy as size grew, which is why they report the gap "does not decrease" with scale.¹ Nothing in that result forbids a future model from closing it. Nothing in it promises one either.
• Assuming a text search covers you. It does not, and the reason is structural: a search finds each hop and still cannot compose them. Grep will hand you the March note and the June note as two separate results, and joining them is your job — which is a real limit on the otherwise-correct advice in Don't Build a RAG Over Your Vault. Grep It.
• Diagnosing composition when retrieval never fired. If the index cannot see the June file at all, no amount of hop-by-hop asking helps. Run the canary test first.
• Writing the note once and never touching it. The synthesis note asserts a conclusion — and conclusions expire. That is why the template ends with a section naming what would overturn it.

Only one of those five is about the model. The other four are about you.

Honest limits: what the synthesis note does not fix

A synthesis note makes the composed answer retrievable in one hop. It does not make the model better at composing. Those are different problems, and only the first one is yours to solve from a text editor. The research is candid about the distance between the two.

The clearest evidence is a null result the IRCoT authors published against their own method: "For GPT3, IRCoT doesn't improve the QA score on IIRC, despite significantly improved retrieval (21 points as shown in Fig. 3)."² Retrieval improved by twenty-one points and the answer did not get better. Better evidence delivery is necessary — it is not sufficient.

Three more limits, stated plainly.

The numbers are borrowed. Press and colleagues measured a set of composed trivia questions; Trivedi and colleagues measured HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC; Tang and Yang measured a news corpus. All three are public-document question answering on models from 2022 to 2024 — the mapping onto a personal vault is an analogy drawn here, not a measurement reported there. There is no multi-hop benchmark for personal notes, and inventing a percentage for one would be worse than having none.

A structured retrieval layer really does win here. That concession is not new; it is the honest half of Everyone's Building a Knowledge Graph for AI. Yours Already Lives in Your Notes. On multi-hop questions specifically, a graph-shaped index outperforms throwing chunks at a model. This post does not re-argue that verdict, and it does not ask you to build one — it offers the remedy that post conceded and skipped, for readers whose vault does not justify a pipeline.

The model may still be wrong about your own life. A retrieved chunk containing both facts is evidence, not truth. The March note may have recorded a decision you later reversed in a conversation you never wrote down — a gap no retrieval strategy can see. Synthesis notes make your record retrievable. They cannot make it complete.

Why a plain Markdown file is the right home for this

A synthesis note is only useful if you can write it in thirty seconds, find it in a year, and read it in ten. That rules out anything living inside a tool's proprietary store — the composed answer you just built would then be reachable only through the same chat box that missed it.

Plain Markdown files answer to every tool you point at them. You can write a synthesis note in any editor, search it with any search, diff it in version control to see when a conclusion changed, and move the folder to another machine.

You can also open it on a laptop that has never heard of the app that made it — no export step, no migration project. When you point a different assistant at your notes next year, the note goes with you unchanged.

There is a quieter benefit. Writing the synthesis note forces you to state the composed answer in your own sentences — which is the moment you find out whether you actually knew it. Half the time the two facts do not join as cleanly as you assumed, and the note you end up writing is a better artifact than the answer you set out to extract.

The file is the durable object. The retrieval pipeline is a rental.

Frequently Asked Questions

Why does my AI miss answers that are in my notes? Because the question you asked names only the first fact. The retriever scores your notes against that question, fetches the note that matches it, and never learns that a second note holds the rest. The model answers fluently from half the evidence, with no signal that anything is missing.

Why can't AI connect information across two notes? Because the second retrieval depends on the first one's result. Trivedi and colleagues describe it as "what to retrieve depends on what has already been derived, which in turn may depend on what was previously retrieved."² A single search pass happens before anything has been derived, so hop two is unknowable at the moment the search runs.

Does a bigger or newer model fix multi-hop questions over my documents? Not on its own, historically. Press and colleagues found single-hop accuracy improved faster than multi-hop accuracy as models scaled, so "the compositionality gap does not decrease."¹ Their own footnote reports a smaller 23.0% gap for GPT-4, with the caveat that the test set may have leaked into its training data.¹

What is the compositionality gap? It is Press and colleagues' term for how often a model answers all sub-questions correctly but still gets the composed question wrong — "a ratio we call the compositionality gap."¹ On GPT-3-family models in 2022 and 2023 they measured it at roughly 40%, constant across model sizes and training techniques.¹

Do I need a knowledge graph to answer multi-hop questions over my notes? At personal scale, no. A structured retrieval layer does measurably better on multi-hop questions, and that concession stands. But the cheap remedies come first: ask hop by hop, then write one note that restates both facts. Build the pipeline only when the file stops being enough.

Will a text search find it if the AI can't? Only partly. Grep will find each hop as a separate result, which is exactly what it is good at. It cannot compose them into an answer, so the join stays manual. Multi-hop breaks keyword search and embedding search in the same way, for the same reason.

How is a synthesis note different from an index or map-of-content note? An index note links out so a human can navigate. A synthesis note restates the facts inside its own body so a single retrieved chunk carries the composed answer. They look alike in an editor and do opposite jobs — one is for walking the vault, the other is for being found in one pass.

Thirty days of this changes what a vault is for. Every question you had to reconstruct by hand becomes a note that answers itself, and the collection stops being a pile of evidence and starts being a set of checkable conclusions. Two facts in two files is a filing decision. Two facts in one file is an answer.

If your notes are plain Markdown files you can rewrite and re-find yourself, that one file is always yours to write — mnmnote.com.