Refresh pages leaderboard

Which LLM keeps a knowledge page right as new memories arrive: takes updates, keeps counts right, and leaves the page alone when nothing changed.

OverviewAll operationsRetainConsolidationReflectRefresh pagesCoding agentsConversations

Pick the LLM that keeps knowledge pages up to date as memories change.

Swapped on this board
LLM
Held fixed
  • Embeddings: bge-small-en-v1.5
  • Reranker: ms-marco-MiniLM-L-6-v2
  • Hindsight: 1.2.0
  • Judge: gemini-3.1-flash-lite

Results

20 cases in 7 categories

Every new memory is read by the LLM. Rows with + decision model also run one; open one to see what each model did and cost.

#ModelCasesLLM callsDecision callsTokens in / out
1
gpt-6-luna
+ hs-decide-1-flash
OpenAI
92.1
90.0
$3.44
36/404.383.720k / 1.2k
2
gemini-3.1-flash-lite
+ hs-decide-1-flash
Google
89.9
90.0
$5.78
36/404.082.517k / 871
3
claude-haiku-5.5
+ hs-decide-1-flash
Anthropic
83.9
82.5
$6.80
33/404.483.732k / 3.4k
4
gemma-4-26b-a4b
+ hs-decide-1-flash
Google · open weights
83.4
77.5
$3.42
31/404.282.722k / 824
5
gpt-6-luna
OpenAI
83.4
82.5
$7.69
33/408.9—77k / 2.2k
6
glm-5.3-flash
+ hs-decide-1-flash
Z.ai · open weights
83.0
77.5
$3.79
31/404.383.722k / 8.8k
7
gemini-3.8-flash
+ hs-decide-1-flash
Google
82.6
90.0
$31
36/403.982.616k / 4.1k
8
glm-5.3-flash
Z.ai · open weights
82.5
75.0
$2.82
30/409.2—85k / 11k
9
gpt-5.6-luna
OpenAI
82.3
85.0
$15
34/408.5—73k / 2.2k
10
claude-haiku-5.5
Anthropic
82.2
85.0
$15
34/409.3—125k / 5.4k
11
gemini-3.1-flash-lite
Google
81.6
85.0
$17
34/407.7—64k / 1.5k
12
deepseek-v4.1-flash
DeepSeek · open weights
79.3
77.5
$8.73
31/4010.3—115k / 12k
13
deepseek-v4.1-flash
+ hs-decide-1-flash
DeepSeek · open weights
77.4
75.0
$9.08
30/405.388.231k / 9.1k
14
gemma-4-26b-a4b
Google · open weights
77.3
72.5
$6.18
29/408.1—75k / 1.5k
15
minimax-m3.1-flash
MiniMax · open weights
75.6
75.0
$14
30/4012.5—161k / 8.3k
16
gpt-6.1-terra
OpenAI
75.0
85.0
$80
34/409.4—87k / 1.4k
17
nemotron-3.5-super-vl
NVIDIA · open weights
73.5
72.5
$15
29/408.2—86k / 10k
18
gemini-3.8-flash
Google
73.4
82.5
$77
33/408.2—74k / 4.9k
19
qwen3.6-35b-a3b
Qwen · open weights
72.7
70.0
$12
28/408.3—94k / 7.0k
20
qwen3.8-27b
Qwen · open weights
71.8
75.0
$33
30/409.6—107k / 12k
21
gpt-5.6-terra
OpenAI
67.1
77.5
$146
31/407.9—72k / 1.9k
22
qwen3.8-flash
Qwen · open weights
64.5
57.5
$11
23/408.8—124k / 9.1k
23
gpt-5.6-sol
OpenAI
59.7
70.0
$240
27/606.3—60k / 974

Click a row for its categories and failed cases. Hover for how a column is computed.

Quality vs. cost across 16 models on Refresh pages

Top left is best value. Hover a point for details.

Best value55%60%65%70%75%80%85%90%95%$5$10$20$50$100$200Cost per 1k cases (log scale) →Quality →gemini-3.1-flash-liteclaude-haiku-5.5gpt-5.6-lunagpt-6.1-terragemini-3.8-flashgpt-6-lunadeepseek-v4.1-flashgpt-5.6-terraminimax-m3.1-flashqwen3.8-27bglm-5.3-flashgemma-4-26b-a4bnemotron-3.5-super-vlgpt-5.6-solqwen3.6-35b-a3bqwen3.8-flash
Model, as its provider’s logoThe same model with hs-decide-1-flash, at the arrow's headBest value: no other model is both cheaper and better

Cost per 1k cases includes the hs-decide-1-flash calls for the runs that use it.

Cases passed by category

Where each model wins and loses. The best in each category is outlined.

ModelPrecisionUpdateAggregationNo overreachStabilityConsistencyOrchestration
gpt-6-luna+hs-decide-1-flash
6/6
8/8
8/8
4/4
4/6
2/4
4/4
gemini-3.1-flash-lite+hs-decide-1-flash
5/6
8/8
7/8
4/4
6/6
2/4
4/4
claude-haiku-5.5+hs-decide-1-flash
6/6
8/8
6/8
4/4
4/6
1/4
4/4
gemma-4-26b-a4b+hs-decide-1-flash
5/6
8/8
4/8
4/4
6/6
0/4
4/4
gpt-6-luna
6/6
7/8
8/8
4/4
4/6
0/4
4/4
glm-5.3-flash+hs-decide-1-flash
6/6
7/8
6/8
4/4
4/6
0/4
4/4
gemini-3.8-flash+hs-decide-1-flash
5/6
8/8
8/8
4/4
6/6
1/4
4/4
glm-5.3-flash
5/6
7/8
6/8
3/4
4/6
1/4
4/4
gpt-5.6-luna
6/6
6/8
8/8
4/4
4/6
2/4
4/4
claude-haiku-5.5
6/6
8/8
6/8
4/4
4/6
2/4
4/4
gemini-3.1-flash-lite
6/6
6/8
5/8
4/4
6/6
3/4
4/4
deepseek-v4.1-flash
5/6
8/8
5/8
4/4
4/6
1/4
4/4
deepseek-v4.1-flash+hs-decide-1-flash
6/6
8/8
4/8
4/4
4/6
0/4
4/4
gemma-4-26b-a4b
6/6
6/8
4/8
4/4
6/6
0/4
3/4
minimax-m3.1-flash
5/6
7/8
5/8
4/4
4/6
1/4
4/4
gpt-6.1-terra
6/6
6/8
8/8
4/4
5/6
1/4
4/4
nemotron-3.5-super-vl
6/6
6/8
6/8
4/4
4/6
0/4
3/4
gemini-3.8-flash
6/6
6/8
5/8
4/4
6/6
2/4
4/4
qwen3.6-35b-a3b
5/6
6/8
5/8
3/4
4/6
2/4
3/4
qwen3.8-27b
6/6
6/8
5/8
4/4
4/6
2/4
3/4
gpt-5.6-terra
6/6
8/8
7/8
4/4
4/6
0/4
2/4
qwen3.8-flash
4/6
7/8
3/8
2/4
3/6
0/4
4/4
gpt-5.6-sol
5/9
7/12
7/12
4/6
2/9
0/6
2/6

Categories

From the evals in this version. Click one for its cases.

  • Precisiona page built across waves keeps the asked attribute apart from its near-misses (3)
    hq-release-prodhq-entity-porthq-numeric-outage
  • Updatea later wave changes the answer: a newer owner, a departed on-call, a renamed service, a corrected figure (4)
    hq-owner-latestdu-oncall-departed-0du-renamed-service-0du-backup-correction-0
  • Aggregationa count grows across waves; a list gains and loses members; a new risk joins the list; a table grows over four deltas (4)
    hq-count-billingdu-enterprise-plan-0du-launch-risk-0du-incident-table-0
  • No overreacha year the bank does not cover; a rule true in one region only (2)
    hq-scoped-2fahq-absent-2024
  • Stabilitya wave with nothing for the page leaves it intact; no page collapses across waves; a history re-recorded every wave is not re-added (3)
    page-never-collapsesdu-quiet-wave-0du-rerecorded-history-0
  • Consistencya changed value changes everywhere the page restates it (English, Chinese) (2)
    pump-maintenance-en-0pump-maintenance-zh-0
  • Orchestration (2)
    fu-escalationfu-release

Also in this version, measured but not graded as cases: test_refresh_pages_benchmark_cost test_refresh_pages_benchmark_cost_all_pages