96.0% accuracy on LongMemEval-S. 480 out of 500 questions correct.
The reader? Kimi K3.1 Pro. Not Claude Opus.
We built a simple memory architecture that reached 96% at 16% of the reader cost of full-context prompting in our control. No vector database. No model-written summaries. No Claude Opus.
ContextStream's latest public result is 96.8% with Claude Opus 5.5. We reached 96.0% with Kimi. Different setups, different protocols, but one thing is clear: you don't need an Opus reader to put a serious memory system in the 96% range.
Disclosure: Wallaby provides inference-as-a-service for curated large language models, including Kimi K3.1 Pro. We ran this evaluation on our own memory-reading setup; the result describes this setup, not customer outcomes.
The result: 480 out of 500
| Item | Value |
|---|---|
| Benchmark | LongMemEval-S, all 500 questions |
| Final correct | 480 |
| Final accuracy | 96.0% |
| Reader | Kimi K3.1 Pro |
| Memory input | One verbatim entry per past session, plus locate and read tools |
| Judge | GPT-4o, official judge prompt |
This is end-to-end answer accuracy, not retrieval@K. Every one of the 500 questions counted, and every answer was judged.
LongMemEval tests whether a chat assistant can answer from a long history of earlier conversations. In the S split, every question comes with its own history: 48 sessions on average in our copy of the cleaned dataset, and a median of about 108K input tokens when we sent it whole to Kimi.
We're proud of the engineering behind this number: a simple memory layout, a strong Kimi reader, and careful cost discipline.
Simple architecture. Serious memory results.
Our approach is straightforward:
- Keep the evidence verbatim. Each past session becomes one dated entry. No model rewrites the history or decides which details might matter later.
- Make reading deliberate. Deterministic tools locate the right entries and read them on demand.
- Skip the extra machinery. No vector database. No model-written summaries. No full-context firehose.
What does that mean for your stack?
- No embedding infrastructure or embedding bill from this architecture
- No model-written summary standing between you and the original evidence
- Full auditability: every answer can be traced back to exact conversation turns
- Fewer moving parts to build, debug and pay for
Good memory infrastructure doesn't have to be mysterious. It has to find the evidence and stay out of the way.
You can inspect our file-based memory rules and audit scripts at wallaby-agent-rules on GitHub.
Want to test this pattern? Use Wallaby as the Kimi reader in your own memory workflow through a standard OpenAI-compatible endpoint. New accounts get $0.50 in free trial credit, with no credit card required. Start with the Quickstart.
Kimi earned its place in the 96% conversation
ContextStream's benchmarks page reports 96.8%, or 484 of 500, with Claude Opus 5.5 as the reader and the official gpt-4o-2024-08-06 judge. It also reports 96.0%, 239 of 249, on the questions it held out from tuning.
Our result came through a different route: Kimi K3.1 Pro, a gpt-4o-2024-11-20 judge snapshot, and the evaluation protocol documented below. We see that as a win for Kimi and for our architecture. Kimi handled long histories, tool calls, cross-session counts, abstentions and preferences, and delivered a 96.0% final result.
You don't need to default to Opus to build a high-scoring memory reader. Kimi has the receipts.
The comparison is not a controlled ranking—the readers, judge snapshots and protocols differ—but it does show what Kimi can do inside a serious memory workflow.
Our memory layer cut reader cost by 84%
This is the result we would put on a slide. We ran the same Kimi model with the whole history pasted into the prompt, no tools, on 162 of the questions.
| Input on the same 162 questions | Correct | Reader cost per question |
|---|---|---|
| Full history in context | 149 | $0.217 |
| Session entries with locate and read tools | 147 | $0.034 |
Same reader. Same questions. 84% lower reader cost. Full context got 7 right that entries missed, and entries got 5 right that full context missed, so we treat the answers as comparable rather than declaring an accuracy winner.
At those rates, running all 500 questions with full context would cost an estimated $109. Our entry setup's first pass cost $17.24. That is the difference between paying to reread everything and paying for the evidence you actually use.
The input difference is stark: a median of 11,405 tokens with our entry setup against 107,567 with full context. And we achieved it with no vector database and no model-written summaries. Simple, inspectable, cheap.
If you're building memory, this is the math that matters: fewer tokens read means lower model bills. Wallaby gives you Kimi through standard OpenAI-compatible endpoints, with prepaid per-token billing and itemized usage. No monthly subscription required. See the docs.
Stop paying to read everything
96.0% with Kimi. 84% lower reader cost in our control. No vector database. No model-written summaries.
We set out to build a memory system that reads less and still answers. On this benchmark, it did both. That is the result we wanted, and we think it's worth celebrating.
The long-memory conversation doesn't need another dogma about bigger models or heavier retrieval stacks. It needs more systems that read the right thing instead of paying to read everything.
Start building yours. New Wallaby accounts get $0.50 in free trial credit—no credit card required. Test Kimi on your own history before you spend a dollar. When you're ready to scale, eligible first top-ups start at $5, and standard top-ups start at $10, with prepaid per-token billing.
For how the same model behaves on Moonshot's own verifier, see our Kimi Vendor Verifier samples.
Evaluation setup and comparability
- **Self-run.** This is our evaluation, not an independent third-party rerun. - **Reader.** `kimi-k3.1-pro`, a fresh conversation per question. Histories were converted mechanically into dated session entries with the original messages verbatim. - **Final score.** Includes two targeted retry rounds on previously missed questions. These used higher reasoning effort, additional tool rounds and instructions written after reviewing earlier errors. Previously correct answers were not rerun. This is not a pre-registered holdout result. - **Evaluation protocol.** Targeting judged misses was a way to control evaluation spend, not a production retry rule. - **Judge.** Official LongMemEval judge prompt and call parameters. The API returned `gpt-4o-2024-11-20`; the official script pins `gpt-4o-2024-08-06`. - **Comparisons and cost.** Full context ran on 162 questions, not all 500. Costs are reader-token estimates at Moonshot's list price, not Wallaby charges or invoices, and exclude the judge. The $109 full-context figure is an estimate from the control; $17.24 is the actual first-pass estimate for our entry setup. - **Top-ups.** The $5 tier applies only to eligible first top-ups. Standard top-ups start at $10.Sources: LongMemEval paper; cleaned dataset and official judge script, downloaded 18 November 2026; ContextStream benchmarks page, read 19 November 2026; our evaluation and full-context runs, 18–19 November 2026, with per-request usage logs and offline tool-call replays.