Model leaderboards

Which model to run with Hindsight: one LLM for everything, the best per operation, and what each setup changes in a real task. Every case is a real Hindsight server, graded by an independent judge.

OverviewAll operations Retain Consolidation Reflect Refresh pagesCoding agents Conversations

Hindsight operations

The best model for each thing Hindsight does, and one LLM behind all of them.

All operations

One LLM behind every Hindsight operation, summed up.

Top every llm operation by score

1
glm-5.3-flash
+ hs-decide-1-flash
Z.ai · open weights
88.7
87% · $7.72
2
gpt-6-luna
+ hs-decide-1-flash
OpenAI
87.6
87% · $9.88
3
claude-haiku-5.5
+ hs-decide-1-flash
Anthropic
85.9
89% · $18
4
deepseek-v4.1-flash
+ hs-decide-1-flash
DeepSeek · open weights
85.3
85% · $12
5
gpt-6-luna
OpenAI
83.9
85% · $16
6
gemini-3.1-flash-lite
+ hs-decide-1-flash
Google
83.8
87% · $22

Retain

Turns raw input into memories.

Top fact extraction by score

1
glm-5.3-flash
Z.ai · open weights
95.0
93% · $0.21
2
minimax-m3.1-flash
MiniMax · open weights
94.6
99% · $0.60
3
gpt-6-luna
OpenAI
94.6
94% · $0.28

Consolidation

Merges facts into observations.

Top llm only by score

1
deepseek-v4.1-flash
+ hs-decide-1-flash
DeepSeek · open weights
87.6
86% · $3.81
2
glm-5.3-flash
+ hs-decide-1-flash
Z.ai · open weights
87.4
84% · $2.90
3
gemma-4-26b-a4b
+ hs-decide-1-flash
Google · open weights
83.0
82% · $5.77

Reflect

Answers questions from memory.

Top agent by score

1
glm-5.3-flash
Z.ai · open weights
93.8
91% · $0.69
2
glm-5.3-flash
+ hs-decide-1-flash
Z.ai · open weights
92.6
94% · $1.48
3
gpt-6-luna
+ hs-decide-1-flash
OpenAI
91.4
94% · $1.94

Refresh pages

Keeps knowledge pages up to date.

Top full delta by score

1
gpt-6-luna
+ hs-decide-1-flash
OpenAI
90.0
90% · $3.16
2
gemini-3.1-flash-lite
+ hs-decide-1-flash
Google
87.4
90% · $5.78
3
gemini-3.8-flash
+ hs-decide-1-flash
Google
82.5
90% · $18

Use cases

Hindsight in a real task, end to end: what each setup changes in practice.

Coding agents

Fixing real bugs with Hindsight’s coding-agent plugin.

SDE-Bench, from the Agent Memory Benchmark

Conversations

Answering questions about long chat histories.

LoCoMo, from the Agent Memory Benchmark

How we test

Hindsight's own system evals, one suite per operation, on Hindsight 1.2.0, against a fresh server through the public API.
Each row swaps one part and keeps the reference setup below. A row marked “with” also runs a decision model.
Graded by gemini-3.1-flash-lite, never the model under test. Each model runs once. Cost comes from the tokens really used, at OpenRouter list prices.

Reference setup

LLM: gemma-4-26b-a4b
Embeddings: bge-small-en-v1.5
Reranker: ms-marco-MiniLM-L-6-v2
Hindsight: 1.2.0
Judge: gemini-3.1-flash-lite