01
Saving something is only the beginning
I want work with agents to accumulate. If a useful connection appears in one session, I want to be able to return to it from another question, without having to remember exactly where it was written. That is the practical concern behind this project.
A memory store can contain both pieces of an answer and still retrieve only one. Imagine asking a question that first requires identifying a person, then finding a fact about them in another document. That second document may look unrelated to the original question. Its relevance only becomes clear after the first step.
I led the framing, implementation, experiments, analysis, and writing of this work with Sean Wu. We tried to separate two problems that can look identical from the outside: the agent cannot reason through the answer, or the retrieval system never gives it the evidence it needs.
02
Separating retrieval from reasoning
We used a difficult subset of MuSiQue, a public question-answering benchmark with annotated two-hop connections. On 159 chains where the query and final document were far apart in embedding space, BM25 and dense retrieval found the final document 0% of the time at recall@10. That means it never appeared among their first ten candidates on this subset.
To check whether retrieval was the obstacle, we ran a separate answer experiment on 80 chains. With standard retrieval, the model answered 5.0% correctly. Supplying the missing final document raised that to 47.5%. The model still made mistakes, but the change localized a substantial part of the failure to the evidence it received.
We also tested stronger retrieval systems on the 159-chain subset. They reached at most 22.6% recall@10, compared with 61.6% for an oracle-assisted condition that resolves the intermediate bridge. I read that comparison as a reason to study how connections are represented and traversed. It is not a claim that every flat retriever must fail, or that the oracle-assisted result is a ready-to-deploy memory system.
03
The benchmark I had to question too
The operational-memory part of the study complicated the story. We mined candidate connections from a real corpus, used a model to judge them, and examined pairs that were distant in embedding space. But distance can mean two things: a subtle connection, or no useful connection at all.
In the later audit, only 2% of candidates in that distant tail were judged genuine bridges. Many apparent connections were artifacts of shared tokens. The audit itself relies on model judgments, with agreement and stability checks; those labels are not infallible ground truth.
This matters to how I present the work. The public MuSiQue experiment supports a retrieval failure on known-valid chains. The operational audit does not establish that the same failure is widespread in everyday agent memory. It shows how easily the process used to find a failure can manufacture examples of it.
04
What I want a memory system to preserve
For someone building a persistent assistant, storage is only part of the job. I would also ask whether the system can recover a connection when the next question uses different language, and whether it can show why that connection is relevant. Those are things to evaluate separately from how many notes it retains.
That is what I mean by recall debt: keeping information without a reliable way to use it later. This study does not measure how that debt grows over time. It gives me a specific retrieval problem to test and a warning about how to construct the test.
I am interested in comparing explicit links, graph retrieval, and learned traversal on validated cases. The next step needs to show which connections those methods recover, what they cost, and which remain missing. My work on handoffs approaches the same practical concern from the other direction: what should we preserve when a piece of work moves to someone new?