Home / Blog / LoCoMo benchmark
Who beats Mem0 on LoCoMo? Lians reports 92.9%.
Lians reports 92.9% judged accuracy at top-200 on the LoCoMo benchmark, compared with Mem0's current published top-200 result of 92.5%. We used Mem0's open-source answer and judge pipeline unchanged. The memory pipelines and retrieval settings differ, so this is an honest comparison of reported results, not a controlled head-to-head claim.
The problem with memory benchmarks
Everyone wins their own benchmark.
Memory benchmark results are usually vendor-run: the vendor picks the retrieval depth, the answer prompt, the judge model, and the grading rubric, then reports one flattering number. The result is a field where every product is somehow first. The fix isn't a better argument; it's giving up control of the grading. mem0 open-sourced the exact harness behind their published LOCOMO results - answer generation, judge prompts, scoring, all of it. So we used that, unmodified, and let their protocol grade us.
The setup
One substitution: the memory layer.
LOCOMO: ten very long multi-session conversations (up to 35 sessions each) with 1,540 graded questions across single-hop, multi-hop, temporal, and open-domain categories - the dataset mem0's own results are built on.
Every conversation turn ingested into Lians with its real timestamp; recall uses the engine's shipped hybrid ranking (semantic + lexical + temporal grounding). The top-k memories per question are dumped in the harness's native format.
The unmodified mem0ai/memory-benchmarks ↗ pipeline reads those dumps and does everything else: gpt-5 writes the answers, gpt-5 judges them, mem0's rubric scores them. We never touch a grade.
LoCoMo results · run 2026-07-09
Lians 92.9% vs Mem0 92.5% at top-200.
| Answer context | Judged accuracy | Mean tokens / question | Share of full context |
|---|---|---|---|
| Lians top-10 | 83.4% | 549 | 3.0% |
| Lians top-20 | 87.3% | 1,083 | 5.9% |
| Lians top-50 | 90.0% | 2,656 | 14.6% |
| Lians top-200 | 92.9% | 10,283 | 56.4% |
| Full conversation in context | - | 18,218 | 100% |
| Mem0 top-200, currently published | 92.5% | their pipeline and retrieval settings | |
What this doesn't say
The caveats, before someone else finds them.
Both reported top-200 results use Mem0's answer and judge pipeline, but each product uses its own memory construction and retrieval pipeline. Treat 92.9 versus 92.5 as a comparison of current published results, not a controlled claim that Lians is universally better. At top-50, Lians reports 90.0% using one-seventh of full-context tokens.
Judge scores vary run to run by a few tenths of a point. We publish the raw evaluation files and per-question outputs, so any number here can be re-judged from the artifacts.
75.0% at top-200. It's in the report, not hidden in an average. We know the failure modes and they're retrieval-side work, not grading disputes.
LoCoMo benchmark FAQ
What the score does and does not prove.
Lians reports 92.9% judged accuracy. Mem0 currently reports 92.5% ↗. The reported difference is 0.4 percentage points.
Yes. Lians exported its retrieved memories into the unmodified mem0ai memory-benchmarks answer and judge pipeline. The underlying memory and retrieval systems remain different.
Yes. The open report includes the protocol, cutoff results, category breakdowns, token counts, artifact paths, and rerun commands.
No. LoCoMo measures long-conversation recall and reasoning. Point-in-time reconstruction, stale-fact suppression, erasure evidence, and lookahead protection require separate tests. See the Lians vs Mem0 comparison →
Why this matters
The compliance engine wins the plain-recall race too.
The standing objection to regulated memory is that the guarantees must cost recall quality - that bitemporal validity windows, deterministic supersession, and audit chains are a tax you pay in accuracy. This run is the counterexample: the same engine that answers "what did the agent know on March 1?" and refuses contaminated recall also beats the best-known general-purpose memory product on the benchmark it popularized. You don't trade recall for defensibility. See both evals side by side →
Reproduce it
Don't trust us. Run it.
git clone https://github.com/Lians-ai/Lians && cd Lians/agentmempython -m benchmarks.locomo_eval --embeddings sentence-transformerspython -m benchmarks.locomo.run --project-name lians --evaluate-only # mem0's harness