Home / Research

Honest recall, measured.

Did the agent answer from what was valid and knowable at the time, and can you prove it afterwards? This eval measures that, live, against real SDKs.

Full write-up ↗Results table →

Definitions

What we measure.

Honest recall

A recall at simulated time T may return only facts whose event time and ingestion time both precede T. Anything else is contamination, however relevant it looks.

Stale fact

A superseded version of a fact (e.g. guidance before a revision) returned as if it were current. Accumulate-everything stores serve these whenever embedding similarity favors the older phrasing.

Lookahead contamination

Two classes: FUTURE_EVENT, where the event hadn't happened at T; and the subtler LATE_REVISION, where the event is old but its corrected figure arrived after T. Vector stores only see event time, so they miss the second class entirely.

Turnkey primitive

Each invariant is scored against a product's public API surface. Either there is an API to call, or there isn't; a thrown invariant is a failure, not a partial.

Method

Five recall checks.

The scenario is a revision chain: NVDA data-center guidance revised upward three times across a quarter, interleaved with unrelated facts. Every system ingests the identical sequence with identical timestamps, then answers the same queries through its own real SDK adapter.

1 · Stale revision suppressed

After a revision lands, the prior figure must not appear in top-5 recall for the natural query. Measures deterministic supersession, not ranking luck.

2 · Point-in-time (as-of) recall

Query the chain as of a date between revisions; the answer must be the figure that was valid then, not today's.

3 · Provable erasure

Erase a data subject; content must become unrecoverable while the audit trail survives, with a certificate to show for it.

4 · Lookahead / backtest guard

A contamination check at a simulated checkpoint must flag both FUTURE_EVENT and LATE_REVISION memories, or refuse recall outright.

5 · Audit-state snapshot at T

Reconstruct exactly what the store contained at a past instant, suitable for handing to an examiner.

Lians executes live against the same engine the server runs; mem0 OSS and Graphiti OSS also execute live in their default documented configs, and the remaining competitors (Letta, Hindsight, Supermemory) run through the same harness via real-SDK adapters. No mocks, no reimplementations of competitors' logic.

Results · last run 2026-07-04

Score the API.

5.0 / 5Lians 2.0 / 5next best (Zep/Graphiti) 0 / 5competitors passing the lookahead guard

Pass = 1, partial = ½. Full per-invariant table and per-competitor analysis on the compare page and in the methodology doc ↗.

Recall quality · LOCOMO · run 2026-07-09

Use the same harness.

92.9%Lians · LLM-judged QA accuracy 91.6%mem0 · their published score 1,986questions · 10 conversations

Lians top-200 retrieval per question, dumped into the unmodified mem0ai/memory-benchmarks ↗ answer + judge pipeline (gpt-5), so grading is the competitor's, verbatim. Lower cutoffs and the full protocol are in the report ↗

Reproduce it

Don't trust us. Run it.

The harness ships in the open-source repo. One command reruns the whole eval; plug in a competitor's API key to score a live column instead of the SDK-surface adapter.

Clonegit clone https://github.com/Lians-ai/Lians && cd Lians/agentmem

Runpython -m benchmarks.compare_regulated

Results on this page are refreshed whenever the harness or a competitor SDK changes; the run date above is the date of the numbers shown, not the page edit date.