Consolidation leaderboard

Which LLM keeps observations current: merges restatements, applies updates, and keeps different things apart, even in large noisy banks.

OverviewAll operationsRetainConsolidationReflectRefresh pagesCoding agentsConversations

Pick the LLM that keeps observations current as new facts arrive: merges restatements, applies updates, keeps different things apart.

Swapped on this board
LLM
Held fixed
  • Embeddings: bge-small-en-v1.5
  • Reranker: ms-marco-MiniLM-L-6-v2
  • Hindsight: 1.2.0
  • Judge: gemini-3.1-flash-lite

Results

25 cases in 9 categories

Every candidate observation goes to the LLM. Rows with + decision model also run one; open one to see what each model did and cost.

#ModelCasesLLM callsDecision callsTokens in / out
1
deepseek-v4.1-flash
+ hs-decide-1-flash
DeepSeek · open weights
87.6
86.0
$3.81
21/255.324.940k / 2.5k
2
glm-5.3-flash
+ hs-decide-1-flash
Z.ai · open weights
87.4
84.0
$2.90
22/255.324.939k / 2.5k
3
gemma-4-26b-a4b
+ hs-decide-1-flash
Google · open weights
83.0
82.0
$5.77
20/255.42545k / 3.0k
4
claude-haiku-5.5
+ hs-decide-1-flash
Anthropic
82.6
84.0
$8.69
21/255.324.954k / 3.4k
5
gemini-3.1-flash-lite
+ hs-decide-1-flash
Google
79.8
82.0
$12
21/255.324.938k / 3.4k
6
minimax-m3.1-flash
MiniMax · open weights
78.4
76.0
$6.35
18/256.7—52k / 4.0k
7
gemini-3.8-flash
+ hs-decide-1-flash
Google
78.3
86.0
$32
22/255.324.938k / 3.2k
8
gemini-3.1-flash-lite
Google
77.4
80.0
$15
21/256.6—49k / 5.3k
9
claude-haiku-5.5
Anthropic
76.5
76.0
$9.72
19/256.6—70k / 5.4k
10
gpt-6-luna
+ hs-decide-1-flash
OpenAI
75.7
70.0
$4.50
16/255.424.939k / 2.1k
11
gemini-3.8-flash
Google
75.6
84.0
$44
21/256.6—49k / 5.0k
12
deepseek-v4.1-flash
DeepSeek · open weights
75.1
68.0
$3.71
16/257.6—59k / 4.0k
13
gpt-6-luna
OpenAI
74.7
68.0
$4.12
18/256.6—50k / 3.2k
14
gemma-4-26b-a4b
Google · open weights
72.0
66.0
$5.52
18/256.8—58k / 4.4k
15
gpt-5.6-luna
OpenAI
71.3
68.0
$8.82
17/256.6—50k / 3.2k
16
qwen3.8-flash
Qwen · open weights
66.0
56.0
$4.41
13/255.2—37k / 4.2k
17
gpt-5.6-terra
OpenAI
65.7
74.0
$85
19/256.6—50k / 3.0k
18
gpt-6.1-terra
OpenAI
64.3
68.0
$45
18/256.6—50k / 4.2k
19
glm-5.3-flash
Z.ai · open weights
55.2
36.0
$2.09
13/2513.8—103k / 4.6k

Click a row for its categories and failed cases. Hover for how a column is computed.

Quality vs. cost across 12 models on Consolidation

Top left is best value. Hover a point for details.

Best value30%40%50%60%70%80%90%$2$5$10$20$50$100Cost per 1k cases (log scale) →Quality →gemini-3.8-flashgemini-3.1-flash-liteclaude-haiku-5.5minimax-m3.1-flashgpt-5.6-terradeepseek-v4.1-flashgpt-5.6-lunagpt-6-lunagpt-6.1-terragemma-4-26b-a4bqwen3.8-flashglm-5.3-flash
Model, as its provider’s logoThe same model with hs-decide-1-flash, at the arrow's headBest value: no other model is both cheaper and better

Cost per 1k cases includes the hs-decide-1-flash calls for the runs that use it.

Cases passed by category

Where each model wins and loses. The best in each category is outlined.

ModelMergeUpdateRecencyCountsEntitiesLanguageMissionScaleRepetition
deepseek-v4.1-flash+hs-decide-1-flash
3/3
3/4
1/2
2/2
2/2
2/2
3/4
3/3
2/3
glm-5.3-flash+hs-decide-1-flash
3/3
4/4
1/2
2/2
1/2
2/2
4/4
3/3
2/3
gemma-4-26b-a4b+hs-decide-1-flash
3/3
4/4
0/2
2/2
2/2
2/2
3/4
2/3
2/3
claude-haiku-5.5+hs-decide-1-flash
2/3
3/4
2/2
2/2
2/2
2/2
3/4
3/3
2/3
gemini-3.1-flash-lite+hs-decide-1-flash
3/3
3/4
1/2
2/2
2/2
2/2
4/4
3/3
1/3
minimax-m3.1-flash
2/3
3/4
2/2
1/2
2/2
1/2
3/4
3/3
1/3
gemini-3.8-flash+hs-decide-1-flash
3/3
3/4
1/2
2/2
2/2
2/2
4/4
3/3
2/3
gemini-3.1-flash-lite
3/3
3/4
1/2
2/2
2/2
2/2
4/4
3/3
1/3
claude-haiku-5.5
1/3
3/4
2/2
2/2
2/2
2/2
3/4
3/3
1/3
gpt-6-luna+hs-decide-1-flash
2/3
3/4
1/2
2/2
1/2
1/2
3/4
3/3
0/3
gemini-3.8-flash
3/3
3/4
2/2
2/2
2/2
2/2
4/4
2/3
1/3
deepseek-v4.1-flash
2/3
3/4
2/2
0/2
2/2
2/2
3/4
2/3
0/3
gpt-6-luna
3/3
3/4
1/2
2/2
1/2
2/2
3/4
3/3
0/3
gemma-4-26b-a4b
3/3
2/4
0/2
2/2
2/2
2/2
3/4
3/3
1/3
gpt-5.6-luna
3/3
3/4
0/2
2/2
1/2
2/2
3/4
3/3
0/3
qwen3.8-flash
3/3
2/4
2/2
1/2
1/2
1/2
3/4
0/3
0/3
gpt-5.6-terra
2/3
3/4
2/2
2/2
2/2
2/2
3/4
2/3
1/3
gpt-6.1-terra
1/3
4/4
2/2
2/2
1/2
2/2
3/4
3/3
0/3
glm-5.3-flash
3/3
2/4
1/2
1/2
1/2
2/2
3/4
0/3
0/3

Categories

From the evals in this version. Click one for its cases.

  • Mergerestatements update one observation; one fact told three ways in one batch is one observation; the same facts under two tags merge in a shared scope (3)
    restatements-mergerepeats-in-one-batchshared-scope-merges-across-tags
  • Updatea new employer, retired tools, a correction, a member leaving a list: the old belief stops reading as current (4)
    employer-supersedesretired-conventionscorrection-retires-false-beliefcascade-to-group
  • Recencyrecency is when a fact was stated, not when it was retained (2)
    recency-is-when-statedrecency-same-day-amendment-arrives-first
  • Countsthe horse farm; no arithmetic on counts (2)
    horse-farmno-arithmetic
  • Entitiesa vague reference resolved by a later fact; lookalike people and numbers stay apart (2)
    resolve-referencedistinct-entities-stay-apart
  • LanguageChinese facts give Chinese observations; a mixed update is rewritten, not patched (2)
    language-chineselanguage-mixed-update
  • Missionwithout a mission chatter is dropped, even in a topic's first batch; a mission can keep ephemera or forbid names (4)
    default-drops-chatternoise-beside-a-routinemission-keeps-ephemeralmission-forbids-names
  • Scale30 near-identical services, 20 lookalike-named people, the services under a mission (3)
    noisy-service-cataloguenoisy-personal-contactsnoisy-service-catalogue-mission
  • Repetitionsteady-state traffic: a coding agent's sessions, an assistant's daily chats, a 12-service bank — mostly restatements and chatter, one rare change; no repeat or small talk becomes an observation (3)
    coding-agent-sessionsassistant-daily-chatsrepetitive-service-catalogue