Graph memory benchmark: LOCOMO and LongMemEval
By Steven Van ·
Open-source Graphify leads LOCOMO recall@10 at 0.497, ties dense RAG on LongMemEval-S and trails one memory layer on LOCOMO QA accuracy.
Open-source Graphify now has published results on two public long-conversation benchmarks, LOCOMO and LongMemEval, along with the harness used to run them. The vendor says the numbers can be rerun rather than taken on trust.
What the benchmarks measure
LOCOMO uses very long multi-session conversations, with questions whose answers sit far back in the history. LongMemEval tests recall, temporal reasoning and knowledge updates across sessions. Recall@10 checks whether the right evidence appears in the top ten results. QA accuracy checks the final answer.
Results
- LOCOMO recall@10 (300 questions): 0.497, against 0.149 and 0.048 for the two memory-layer systems tested.
- LOCOMO QA accuracy (300 questions): 45.3%. One memory layer scored 49.7% and the other 27.3%.
- LongMemEval-S QA accuracy (50 questions): 76%, tied with dense RAG.
Building the index used zero LLM credits. Graphify leads on LOCOMO retrieval, ties on LongMemEval-S and trails one memory-layer system on LOCOMO QA accuracy. The vendor says retrieval and answering are separate steps: the graph plus hybrid retrieval puts the right evidence in the top ten more often, but turning that evidence into a correct answer also depends on the answering prompt and model.
How it was run
Every system ran as an adapter in one harness, with one shared model (Kimi K2.6), the same token budgets and the same grader. The judge was blind-validated against a second independent judge at 90.6% agreement, with a Cohen's kappa of 0.81. BENCHMARKS.md in the open-source repository lists the commands. The vendor cautions that 50 LongMemEval questions are enough to show a tie but not to separate close systems.
Measuring token savings on your own repository
Running graphify benchmark in a built repository compares the tokens needed to read the whole corpus with the tokens a graph query returns for a fixed set of sample questions. It estimates context savings and does not test accuracy, and the ratio grows with repository size. Setup is uv tool install graphifyy, then graphify update ., then graphify benchmark.
- itsdangerous: 372 nodes and 695 edges, about 24,800 tokens naive against about 3,205 per query, 7.7x fewer.
- Flask: 2,341 nodes and 3,753 edges, about 156,066 tokens naive against about 1,234 per query, 126.5x fewer.
