Reflect leaderboard

Which LLM answers best from memory: uses the latest value, aggregates correctly, and says so when the answer is not there.

OverviewAll operationsRetainConsolidationReflectRefresh pagesCoding agentsConversations

Pick the LLM that answers questions from memory, planning its own searches and deciding when it has enough.

Swapped on this board
LLM
Held fixed
  • Embeddings: bge-small-en-v1.5
  • Reranker: ms-marco-MiniLM-L-6-v2
  • Hindsight: 1.2.0
  • Judge: gemini-3.1-flash-lite

Results

17 cases in 7 categories

The LLM plans its own searches and decides when it has enough. Rows with + decision model also run one; open one to see what each model did and cost.

#ModelCasesLLM callsDecision callsTokens in / out
1
glm-5.3-flash
Z.ai · open weights
93.8
91.2
$0.69
15/174.5—44k / 652
2
glm-5.3-flash
+ hs-decide-1-flash
Z.ai · open weights
92.6
94.1
$1.48
16/172.615.88.8k / 494
3
gpt-6-luna
+ hs-decide-1-flash
OpenAI
91.4
94.1
$1.94
16/172.615.68.8k / 129
4
gpt-6-luna
OpenAI
88.3
94.1
$3.98
16/174.4—40k / 235
5
claude-haiku-5.5
+ hs-decide-1-flash
Anthropic
87.9
91.2
$2.69
15/172.515.513k / 489
6
deepseek-v4.1-flash
DeepSeek · open weights
87.5
88.2
$1.82
15/174.4—43k / 1.1k
7
claude-haiku-5.5
Anthropic
86.5
94.1
$5.98
16/174.3—56k / 663
8
gpt-5.6-luna
OpenAI
85.7
94.1
$7.23
16/174.3—37k / 268
9
gemma-4-26b-a4b
+ hs-decide-1-flash
Google · open weights
85.5
85.3
$1.84
16/172.615.59.0k / 217
10
minimax-m3.1-flash
MiniMax · open weights
85.4
91.2
$4.87
16/175.0—64k / 1.3k
11
deepseek-v4.1-flash
+ hs-decide-1-flash
DeepSeek · open weights
85.3
85.3
$1.91
14/172.615.99.5k / 772
12
gemma-4-26b-a4b
Google · open weights
85.0
88.2
$3.25
15/174.2—41k / 323
13
gemini-3.1-flash-lite
Google
84.7
94.1
$9.01
16/174.2—41k / 348
14
gemini-3.8-flash
+ hs-decide-1-flash
Google
83.2
91.2
$7.91
16/172.414.87.5k / 176
15
gemini-3.1-flash-lite
+ hs-decide-1-flash
Google
83.1
85.3
$3.15
14/172.4157.7k / 188
16
gemini-3.8-flash
Google
78.7
94.1
$36
16/174.4—45k / 438
17
qwen3.8-flash
Qwen · open weights
77.6
79.4
$4.31
15/174.9—64k / 797
18
gpt-6.1-terra
OpenAI
77.5
94.1
$47
16/174.3—46k / 230
19
gpt-5.6-terra
OpenAI
75.5
94.1
$76
16/174.4—39k / 273

Click a row for its categories and failed cases. Hover for how a column is computed.

Quality vs. cost across 12 models on Reflect

Top left is best value. Hover a point for details.

Best value78%80%82%84%86%88%90%92%94%96%$1$2$5$10$20$50Cost per 1k cases (log scale) →Quality →gemini-3.1-flash-litegemini-3.8-flashclaude-haiku-5.5gpt-5.6-lunagpt-5.6-terragpt-6-lunagpt-6.1-terraminimax-m3.1-flashglm-5.3-flashgemma-4-26b-a4bdeepseek-v4.1-flashqwen3.8-flash
Model, as its provider’s logoThe same model with hs-decide-1-flash, at the arrow's headBest value: no other model is both cheaper and better

Cost per 1k cases includes the hs-decide-1-flash calls for the runs that use it.

Cases passed by category

Where each model wins and loses. The best in each category is outlined.

ModelPrecisionUpdateAggregationNo overreachSource authorityPagesOrchestration
glm-5.3-flash
3/3
2/2
1/2
2/2
3/4
2/2
2/2
glm-5.3-flash+hs-decide-1-flash
3/3
2/2
2/2
2/2
3/4
2/2
2/2
gpt-6-luna+hs-decide-1-flash
3/3
2/2
2/2
2/2
3/4
2/2
2/2
gpt-6-luna
3/3
2/2
2/2
2/2
3/4
2/2
2/2
claude-haiku-5.5+hs-decide-1-flash
3/3
2/2
2/2
2/2
2/4
2/2
2/2
deepseek-v4.1-flash
3/3
2/2
2/2
2/2
2/4
2/2
2/2
claude-haiku-5.5
3/3
2/2
2/2
2/2
3/4
2/2
2/2
gpt-5.6-luna
3/3
2/2
2/2
2/2
3/4
2/2
2/2
gemma-4-26b-a4b+hs-decide-1-flash
3/3
2/2
2/2
2/2
3/4
2/2
2/2
minimax-m3.1-flash
3/3
2/2
2/2
2/2
3/4
2/2
2/2
deepseek-v4.1-flash+hs-decide-1-flash
3/3
2/2
1/2
2/2
3/4
1/2
2/2
gemma-4-26b-a4b
3/3
2/2
1/2
2/2
3/4
2/2
2/2
gemini-3.1-flash-lite
3/3
2/2
2/2
2/2
3/4
2/2
2/2
gemini-3.8-flash+hs-decide-1-flash
3/3
2/2
2/2
2/2
3/4
2/2
2/2
gemini-3.1-flash-lite+hs-decide-1-flash
3/3
2/2
1/2
2/2
3/4
1/2
2/2
gemini-3.8-flash
3/3
2/2
2/2
2/2
3/4
2/2
2/2
qwen3.8-flash
3/3
2/2
2/2
2/2
3/4
1/2
2/2
gpt-6.1-terra
3/3
2/2
2/2
2/2
3/4
2/2
2/2
gpt-5.6-terra
3/3
2/2
2/2
2/2
3/4
2/2
2/2

Categories

From the evals in this version. Click one for its cases.

  • Precisionthe answer differs from its near-misses by one attribute: a version, a service name, a number (3)
    hq-release-prodhq-entity-porthq-numeric-outage
  • Updatewhat is true now, when the bank also holds what was true before (2)
    hq-owner-latestsp-rotation
  • Aggregationa count across many facts; a rule together with its approved exception (2)
    hq-count-billingsp-friday-deploy
  • No overreacha year the bank does not cover; a rule true in one region only (2)
    hq-scoped-2fahq-absent-2024
  • Source authoritythe handbook against confident chatter; questions only one source answers (4)
    sp-datastoresp-approvalssp-postmortemsp-upgrade-owner
  • Pagesthe bank holds a page about the question, and the answer comes from it (2)
    hq-release-prodhq-count-billing
  • Orchestration (2)
    fu-escalationfu-release

Also in this version, measured but not graded as cases: test_reflect_benchmark_coding_agent_decision_model