All operations leaderboard

Most setups run one LLM for everything, sometimes with a decision model. This board sums up the four operation boards that use an LLM: quality is the mean over them, cost is what running each of them costs. No runs of its own.

OverviewAll operationsRetainConsolidationReflectRefresh pagesCoding agentsConversations

Pick one LLM for all of Hindsight. Each row is an LLM, alone or with a decision model, run through every operation that uses an LLM; its score sums them up.

Swapped on this board
LLM
Held fixed
  • Embeddings: bge-small-en-v1.5
  • Reranker: ms-marco-MiniLM-L-6-v2
  • Hindsight: 1.2.0
  • Judge: gemini-3.1-flash-lite

Results

104 cases in 4 operations

A row needs a run of every operation. A row with a decision model takes its LLM's retain run, since retain does not use one.

#ModelCasesLLM callsDecision callsTokens in / out
1
glm-5.3-flash
+ hs-decide-1-flash
Z.ai · open weights
88.7
87.1
$7.72
183/2083.524.718k / 1.3k
2
gpt-6-luna
+ hs-decide-1-flash
OpenAI
87.6
87.0
$9.88
182/2083.124.617k / 793
3
claude-haiku-5.5
+ hs-decide-1-flash
Anthropic
85.9
88.5
$18
187/2083.124.624k / 1.6k
4
deepseek-v4.1-flash
+ hs-decide-1-flash
DeepSeek · open weights
85.3
85.1
$12
181/2083.425.519k / 1.6k
5
gpt-6-luna
OpenAI
83.9
84.7
$16
178/2084.6—35k / 1.2k
6
gemini-3.1-flash-lite
+ hs-decide-1-flash
Google
83.8
86.6
$22
181/2083.124.315k / 1.3k
7
deepseek-v4.1-flash
DeepSeek · open weights
83.7
81.9
$11
174/2085.2—45k / 2.2k
8
claude-haiku-5.5
Anthropic
83.2
87.9
$31
185/2084.7—53k / 2.4k
9
gemma-4-26b-a4b
+ hs-decide-1-flash
Google · open weights
82.7
81.1
$12
168/2083.224.519k / 1.2k
10
minimax-m3.1-flash
MiniMax · open weights
82.4
85.2
$24
182/2085.5—56k / 2.6k
11
gemini-3.8-flash
+ hs-decide-1-flash
Google
82.2
90.6
$61
190/2083.024.315k / 1.2k
12
glm-5.3-flash
Z.ai · open weights
81.7
73.8
$4.58
157/2086.7—51k / 2.0k
13
gemini-3.1-flash-lite
Google
81.3
87.1
$42
181/2084.4—32k / 1.9k
14
gpt-5.6-luna
OpenAI
81.2
85.0
$31
178/2084.5—34k / 1.3k
15
gemma-4-26b-a4b
Google · open weights
78.3
76.6
$16
159/2084.5—37k / 1.7k
16
gemini-3.8-flash
Google
77.2
89.0
$147
187/2084.5—35k / 1.9k
17
qwen3.8-flash
Qwen · open weights
75.2
72.6
$17
160/2084.4—45k / 2.0k
18
gpt-6.1-terra
OpenAI
74.6
86.2
$173
182/2084.7—38k / 1.4k
19
gpt-5.6-terra
OpenAI
70.7
84.3
$307
177/2084.4—34k / 1.2k

Click a row for its categories and failed cases. Hover for how a column is computed.

Quality vs. cost across 12 models on All operations

Top left is best value. Hover a point for details.

Best value70%72%74%76%78%80%82%84%86%88%90%92%$5$10$20$50$100$200Cost per 1k cases (log scale) →Quality →gemini-3.8-flashclaude-haiku-5.5gemini-3.1-flash-litegpt-6.1-terraminimax-m3.1-flashgpt-5.6-lunagpt-6-lunagpt-5.6-terradeepseek-v4.1-flashgemma-4-26b-a4bglm-5.3-flashqwen3.8-flash
Model, as its provider’s logoThe same model with hs-decide-1-flash, at the arrow's headBest value: no other model is both cheaper and better

Cost per 1k cases includes the hs-decide-1-flash calls for the runs that use it.

Cases passed by category

Where each model wins and loses. The best in each category is outlined.

ModelRetainConsolidationReflectRefresh pages
glm-5.3-flash+hs-decide-1-flash
78/84
42/50
32/34
31/40
gpt-6-luna+hs-decide-1-flash
79/84
35/50
32/34
36/40
claude-haiku-5.5+hs-decide-1-flash
81/84
42/50
31/34
33/40
deepseek-v4.1-flash+hs-decide-1-flash
79/84
43/50
29/34
30/40
gpt-6-luna
79/84
34/50
32/34
33/40
gemini-3.1-flash-lite+hs-decide-1-flash
75/84
41/50
29/34
36/40
deepseek-v4.1-flash
79/84
34/50
30/34
31/40
claude-haiku-5.5
81/84
38/50
32/34
34/40
gemma-4-26b-a4b+hs-decide-1-flash
67/84
41/50
29/34
31/40
minimax-m3.1-flash
83/84
38/50
31/34
30/40
gemini-3.8-flash+hs-decide-1-flash
80/84
43/50
31/34
36/40
glm-5.3-flash
78/84
18/50
31/34
30/40
gemini-3.1-flash-lite
75/84
40/50
32/34
34/40
gpt-5.6-luna
78/84
34/50
32/34
34/40
gemma-4-26b-a4b
67/84
33/50
30/34
29/40
gemini-3.8-flash
80/84
42/50
32/34
33/40
qwen3.8-flash
82/84
28/50
27/34
23/40
gpt-6.1-terra
82/84
34/50
32/34
34/40
gpt-5.6-terra
77/84
37/50
32/34
31/40