Wallaby.Dispatch

Why your AI agent's memory says it saved but didn't

2026-10-18 by the Wallaby Dispatch editorial team Playbook memory agents-md coding-agents context-engineering
Reviewed against primary sources; Wallaby numbers come from our own production gateway logs.

On October 13, an agentmemory user filed an issue with a code block that should not exist. Two memory_save calls, two returned IDs, and about six days later, no memories. The engine had hung on October 7. It held its listening sockets open, never accepted a connection, and the client kept confirming saves that never reached disk. Call it the state-assertion bug: a system's report about its own state diverges from the state.

It is not a retrieval failure. We think it is the most expensive failure class in agent memory, because it corrupts the one asset you need to recover from every other failure, the record of what happened, and it does so one green check at a time.

This post takes three threads of that class from agentmemory, an open-source memory system for coding agents with about 29,600 stars, and pairs them with the four escalating fixes we ran on the same class inside our own memory discipline, over two days in October. The short version: a status claim is not evidence. Make the claim name the check that would prove it, make the check read the live system rather than the record of it, and make the wrong thing impossible before you make it visible.

Disclosure: Wallaby Token sells inference for open-weight models, and we keep our own project records in a file-based memory discipline. We have a stake in this layer, so check the receipts and the links.

A receipt from a till the store never reads

The issue is titled "hung agentmemory was not saving but giving ids," and the body is nearly boring, which is what makes it useful. The reporter ran two saves and pasted the responses:

{"saved":"mem_mv6tghgk_02a5a2f023b8"}
{"saved":"mem_mv6tghzw_4aba68f9c6f7"}

A later session could not find either one. The reporter's diagnosis: the engine had been hung for about six days, holding its listeners but never accepting, so the UI showed zero memories while both saves returned IDs for rows that never touched disk.

The follow-up pull request explains the mechanism better than the report does. When the health check or a proxied call fails, the MCP shim writes the memory to a local fallback file, ~/.agentmemory/standalone.json, and replies {"saved": id}. Nothing copies that file to the server. The fallback is announced only on stderr.

That last sentence is the whole story. The save did happen. The receipt was accurate about the ID and silent about the address, and the address is the only part that matters, because a memory in a file the server never reads is a memory you will never recall.

The open patch does not prevent the loss. It prevents the lie: fallback saves and deletes would carry storage: "local" and a warning that the server did not confirm the call. A patch that makes a failure legible is worth more than a patch that makes it rare, because the rare one still happens on a day you are not looking.

Not saving, or saving somewhere else?

That pull request also shows the second form of the same bug: a labeling problem wearing a storage costume. The user's mental state is "it is not saving." The system's state is "it is saving somewhere the server does not read." Both sentences are true, and the interface offers one word for three different outcomes: saved, queued, and gone.

This shape is everywhere: an offline write queue, a mock returning fixtures, a cache serving stale values. The write path is not wrong. The vocabulary is.

The test we now apply to any status field is whether it can be false. A field that always reads saved is decoration, not a status. If the server did not confirm, the response should say the server did not confirm, and it should name the store that actually holds the bytes. The patch here does exactly that, and the interesting part is how small it is. One field and one warning, in the place where the user is already looking.

Contrast that with the original signal. The fallback was announced on stderr, which is where truthful information goes to be ignored. A warning that requires a human to tail a log during an agent loop is a note in a drawer.

success: true after the data is gone

On October 16, a second report took the same shape through a different door. POST /agentmemory/import with strategy: "replace" deletes every existing scope before it checks the optional sections. Send lessons as an empty object, and the response is success: true with every lesson gone. Send it as a string, and the call throws after the deletes. Either way the store is empty by the time anyone finds out.

The proposed fix validates before deleting. That is the strongest shape in this post, because it does not depend on anyone reading a warning at the right moment. A malformed section is rejected before anything is lost.

Why do tests miss this? Because each step is correct alone. Replace does delete scopes. The validator does reject malformed sections. The handler does return success. The bug is the order, and order rarely gets its own test until someone has been burned by it.

Why is this class so expensive?

Retrieval errors announce themselves. You ask for a fact, get the wrong one, and notice. A state-report error waits. You build the next three steps on it, and when something finally breaks, you debug using a record that is already false.

Then comes the calibration cost. Every time the interface says fine and it isn't, you pay twice: once for the lost write, and once for the habit of not checking. That habit is the expensive part, and nothing in a dashboard measures it.

The benchmark layer does not cover this. agentmemory publishes 95.2% R@5 on LongMemEval-S from its own run, reproducible from the harness in the repo. That number is real, and it says nothing about whether a write reached the store. Retrieval quality and write truthfulness are different contracts. A system can be excellent at the first and still fail the second on a Tuesday, which is roughly what happened here.

None of this is an argument against the project. The tracker is public, the fix proposals are public, and the reports are specific enough to reproduce. The interesting fact is that a team this careful about evaluation still shipped the failure. Carelessness is the wrong explanation. The cause is a status message that promises more than the system can know.

The four rounds we ran on the same class

We recognized the shape because we have been losing to it for months. Our own memory discipline makes claims about a live project: what shipped, what is still in flight, what was decided. Four times in two days in October, one of those claims was wrong, and each fix was a stronger shape than the last.

The first fix added a question to our instructions. Before stating a status, ask whether this session actually observed it. Within a day we produced two wrong status sentences, one built on a snapshot that was twenty-five days old and one that called an item pending after an exception had already closed it. A question in a document is a reminder, and reminders are read by the same mind that is about to be wrong.

A script replaced the question the next day. It reads five sources in a fixed priority order, verdicts first and stale plans last, and prints which source says what. It also added a write-back rule: when an item closes, the plan line gets a pointer. The script was correct. What failed was the installation. We recorded that the check ran before status sentences, and the file that would have told anyone to run it never contained that line. The gate existed in the report about the gate.

Then came a machine light. A daily scan flags log entries that contain status words without an evidence anchor. That scan found real problems, and it still missed the failure that mattered, because it scanned the log while the failure happened in a sentence about to be spoken in a conversation. The sensor was mounted in the archive while the leak was in the room.

The fourth fix moved the check into the generator. A live mode reads the draft's front matter, the built output, and the live page before a status claim can be written, and the drafting template has a field for the evidence that cannot be left blank. The class recurred afterward in a sentence rather than a document. We count that as evidence for the ranking below, not against the check, because a check only covers the surface it is mounted on.

Fix shapes, ranked by when they act

Shape When it acts What it gives you Example
Make it impossible Before the action The bad state cannot occur Validate before delete (#1576)
Wire it to the moment At the action The reader sees the truth at the moment of choice Response says storage: "local" (#1563)
Detect after the fact After the action You can find it later, if you look Daily scan for unevidenced claims
Remember to remember Depends on the moment Nothing you can rely on Our self-check fixes: 0 for 4

Rank the shapes by where the failure can enter, not by how much effort the fix takes. The weakest shape is also the one that feels most like progress, because writing a rule feels like doing something, and the day you write it is usually the day the last failure is still fresh.

Across our four rounds, the first three rules were all true. The question was right, the script was right, the scan was right. They failed because they lived in a channel the decision maker does not visit at the moment of decision. That is the same defect as a fallback announced on stderr, and it is why we now rank fixes by when they act.

The checks that hold

Our rule now has two halves. A positive claim names the check that would prove it, and that check reads the live system, not the record of the live system. A negative claim, the kind that says not done, does not exist, or cannot be done, has to answer three questions before it leaves the desk:

  1. Which source I have not checked is most likely to hold the counterexample?
  2. What evidence would overturn this conclusion, and did I look for it?
  3. Which related fact did I already see, and then ignore?

These questions are mechanical on purpose. They are the cheapest substitute we have for a machine that cannot be lied to, and they work best when the answer has to name a command and its output.

Four things to copy on Monday:

  • Read the write back. memory_save returns an ID. Recall that ID in a fresh session and assert the content, not the status code. One extra call buys you the only evidence that counts.
  • Label the fallback. If any client path can store locally, its response must say so in the result, not in stderr. If the only signal is a log line, you have no signal.
  • Order destructive operations. Validate, then delete, and test the malformed input path, because that is the branch where the store disappears.
  • Count your fixes. If the same class fails twice, change the shape. On the third failure, move the check into the tool that produces the claim, not into the notes around it.

What we have not solved

Our failure registry is still event driven and written by a human, which means the judgment stays human and the coverage depends on someone noticing. The live check reads the post source, the build output, and the live URL. It does not read the rest of what we claim about our own work. The patches here are mostly reactive in the same way: the fallback still writes to a file the server never reads, and the open patch makes it say so.

There is a specification question underneath, and we would like to see it taken seriously. When a memory API cannot confirm that a write reached durable storage, should the response say saved at all? Accepted, queued, and confirmed are three different promises, and most memory APIs collapse them into one. The first vendor to publish that distinction in its contract will make this entire bug class cheaper to avoid, for everyone, including the systems that already handle it well.

Until then, the discipline is the same one we apply to our own records: treat every status message as a claim, and check the claim against the live system before you build on it. The memory files we use are open source in wallaby-agent-rules, and the failure registry behind this post is internal, which is the next asymmetry worth closing.

Sources and further reading

Like the Dispatch? Add Wallaby as a preferred source on Google, or subscribe via RSS.
Did this help?