Reflect leaderboard

Which LLM answers best from memory: uses the latest value, aggregates correctly, and says so when the answer is not there.

OverviewAll operationsRetainConsolidationReflectRefresh pagesCoding agentsConversations

Pick the LLM that answers questions from memory, planning its own searches and deciding when it has enough.

Swapped on this board
LLM
Held fixed
  • Embeddings: bge-small-en-v1.5
  • Reranker: ms-marco-MiniLM-L-6-v2
  • Hindsight: 1.2.0
  • Judge: gemini-3.1-flash-lite

Results

17 cases in 7 categories

The LLM plans its own searches and decides when it has enough. Rows with + decision model also run one; open one to see what each model did and cost.

#ModelCasesLLM callsDecision callsTokens in / out
1
glm-5.3-flash
Z.ai · open weights
93.8
91.2
$0.91
31/344.5—44k / 2.0k
2
glm-5.3-flash
+ hs-decide-1-flash
Z.ai · open weights
93.5
94.1
$1.56
32/342.615.88.8k / 1.4k
3
gpt-6-luna
+ hs-decide-1-flash
OpenAI
92.3
94.1
$2.05
32/342.615.68.8k / 345
4
gpt-6-luna
OpenAI
89.3
94.1
$4.12
32/344.4—40k / 511
5
claude-haiku-5.5
+ hs-decide-1-flash
Anthropic
88.9
91.2
$2.84
31/342.515.513k / 790
6
deepseek-v4.1-flash
DeepSeek · open weights
87.8
88.2
$2.25
30/344.4—43k / 2.0k
7
claude-haiku-5.5
Anthropic
87.6
94.1
$6.16
32/344.3—56k / 1.0k
8
gemma-4-26b-a4b
+ hs-decide-1-flash
Google · open weights
86.7
85.3
$1.84
29/342.615.59.0k / 217
9
gpt-5.6-luna
OpenAI
86.7
94.1
$7.46
32/344.3—37k / 458
10
gemma-4-26b-a4b
Google · open weights
86.2
88.2
$3.25
30/344.2—41k / 323
11
minimax-m3.1-flash
MiniMax · open weights
86.1
91.2
$5.37
31/345.0—64k / 2.2k
12
gemini-3.1-flash-lite
Google
85.9
94.1
$9.01
32/344.2—41k / 348
13
qwen3.8-27b
Qwen · open weights
85.8
94.1
$9.29
32/344.4—50k / 2.0k
14
deepseek-v4.1-flash
+ hs-decide-1-flash
DeepSeek · open weights
85.6
85.3
$2.34
29/342.615.99.5k / 1.6k
15
qwen3.6-35b-a3b
Qwen · open weights
85.5
91.2
$6.14
31/344.6—51k / 3.1k
16
gemini-3.1-flash-lite
+ hs-decide-1-flash
Google
84.3
85.3
$3.15
29/342.4157.7k / 188
17
gemini-3.8-flash
+ hs-decide-1-flash
Google
83.1
91.2
$11
31/342.414.87.5k / 853
18
nemotron-3.5-super-vl
NVIDIA · open weights
82.9
88.2
$7.03
30/344.5—51k / 1.8k
19
gemini-3.8-flash
Google
79.5
94.1
$40
32/344.4—45k / 1.4k
20
gpt-6.1-terra
OpenAI
78.7
94.1
$47
32/344.3—46k / 306
21
qwen3.8-flash
Qwen · open weights
78.0
79.4
$5.21
27/344.9—64k / 2.7k
22
gpt-5.6-terra
OpenAI
76.6
94.1
$77
32/344.4—39k / 425
23
gpt-5.6-sol
OpenAI
72.8
94.1
$183
31/514.4—44k / 360

Click a row for its categories and failed cases. Hover for how a column is computed.

Quality vs. cost across 16 models on Reflect

Top left is best value. Hover a point for details.

Best value78%80%82%84%86%88%90%92%94%96%$1$2$5$10$20$50$100$200Cost per 1k cases (log scale) →Quality →gpt-5.6-solgemini-3.1-flash-litegemini-3.8-flashclaude-haiku-5.5gpt-5.6-lunagpt-5.6-terragpt-6-lunagpt-6.1-terraqwen3.8-27bminimax-m3.1-flashqwen3.6-35b-a3bglm-5.3-flashgemma-4-26b-a4bdeepseek-v4.1-flashnemotron-3.5-super-vlqwen3.8-flash
Model, as its provider’s logoThe same model with hs-decide-1-flash, at the arrow's headBest value: no other model is both cheaper and better

Cost per 1k cases includes the hs-decide-1-flash calls for the runs that use it.

Cases passed by category

Where each model wins and loses. The best in each category is outlined.

ModelPrecisionUpdateAggregationNo overreachSource authorityPagesOrchestration
glm-5.3-flash
6/6
4/4
3/4
4/4
6/8
4/4
4/4
glm-5.3-flash+hs-decide-1-flash
6/6
4/4
4/4
4/4
6/8
4/4
4/4
gpt-6-luna+hs-decide-1-flash
6/6
4/4
4/4
4/4
6/8
4/4
4/4
gpt-6-luna
6/6
4/4
4/4
4/4
6/8
4/4
4/4
claude-haiku-5.5+hs-decide-1-flash
6/6
4/4
4/4
4/4
5/8
4/4
4/4
deepseek-v4.1-flash
6/6
4/4
4/4
4/4
4/8
4/4
4/4
claude-haiku-5.5
6/6
4/4
4/4
4/4
6/8
4/4
4/4
gemma-4-26b-a4b+hs-decide-1-flash
6/6
4/4
3/4
4/4
5/8
3/4
4/4
gpt-5.6-luna
6/6
4/4
4/4
4/4
6/8
4/4
4/4
gemma-4-26b-a4b
6/6
4/4
2/4
4/4
6/8
4/4
4/4
minimax-m3.1-flash
6/6
4/4
4/4
3/4
6/8
4/4
4/4
gemini-3.1-flash-lite
6/6
4/4
4/4
4/4
6/8
4/4
4/4
qwen3.8-27b
6/6
4/4
4/4
4/4
6/8
4/4
4/4
deepseek-v4.1-flash+hs-decide-1-flash
6/6
4/4
2/4
4/4
6/8
3/4
4/4
qwen3.6-35b-a3b
6/6
4/4
4/4
4/4
6/8
3/4
4/4
gemini-3.1-flash-lite+hs-decide-1-flash
6/6
4/4
2/4
4/4
6/8
3/4
4/4
gemini-3.8-flash+hs-decide-1-flash
6/6
4/4
3/4
4/4
6/8
4/4
4/4
nemotron-3.5-super-vl
6/6
4/4
4/4
3/4
5/8
4/4
4/4
gemini-3.8-flash
6/6
4/4
4/4
4/4
6/8
4/4
4/4
gpt-6.1-terra
6/6
4/4
4/4
4/4
6/8
4/4
4/4
qwen3.8-flash
4/6
4/4
2/4
4/4
6/8
3/4
4/4
gpt-5.6-terra
6/6
4/4
4/4
4/4
6/8
4/4
4/4
gpt-5.6-sol
6/9
4/6
4/6
4/6
6/12
4/6
3/6

Categories

From the evals in this version. Click one for its cases.

  • Precisionthe answer differs from its near-misses by one attribute: a version, a service name, a number (3)
    hq-release-prodhq-entity-porthq-numeric-outage
  • Updatewhat is true now, when the bank also holds what was true before (2)
    hq-owner-latestsp-rotation
  • Aggregationa count across many facts; a rule together with its approved exception (2)
    hq-count-billingsp-friday-deploy
  • No overreacha year the bank does not cover; a rule true in one region only (2)
    hq-scoped-2fahq-absent-2024
  • Source authoritythe handbook against confident chatter; questions only one source answers (4)
    sp-datastoresp-approvalssp-postmortemsp-upgrade-owner
  • Pagesthe bank holds a page about the question, and the answer comes from it (2)
    hq-release-prodhq-count-billing
  • Orchestration (2)
    fu-escalationfu-release

Also in this version, measured but not graded as cases: test_reflect_benchmark_coding_agent_decision_model