Pick the LLM that answers questions from memory, planning its own searches and deciding when it has enough.
- Embeddings: bge-small-en-v1.5
- Reranker: ms-marco-MiniLM-L-6-v2
- Hindsight: 1.2.0
- Judge: gemini-3.1-flash-lite
Results
17 cases in 7 categoriesThe LLM plans its own searches and decides when it has enough. Rows with + decision model also run one; open one to see what each model did and cost.
| # | Model | Cases | LLM calls | Decision calls | Tokens in / out | |||
|---|---|---|---|---|---|---|---|---|
| 1 | ![]() glm-5.3-flash Z.ai · open weights | 93.8 | 91.2 | $0.91 | 31/34 | 4.5 | — | 44k / 2.0k |
| 2 | ![]() glm-5.3-flash + hs-decide-1-flash Z.ai · open weights | 93.5 | 94.1 | $1.56 | 32/34 | 2.6 | 15.8 | 8.8k / 1.4k |
| 3 | ![]() gpt-6-luna + hs-decide-1-flash OpenAI | 92.3 | 94.1 | $2.05 | 32/34 | 2.6 | 15.6 | 8.8k / 345 |
| 4 | ![]() gpt-6-luna OpenAI | 89.3 | 94.1 | $4.12 | 32/34 | 4.4 | — | 40k / 511 |
| 5 | ![]() claude-haiku-5.5 + hs-decide-1-flash Anthropic | 88.9 | 91.2 | $2.84 | 31/34 | 2.5 | 15.5 | 13k / 790 |
| 6 | ![]() deepseek-v4.1-flash DeepSeek · open weights | 87.8 | 88.2 | $2.25 | 30/34 | 4.4 | — | 43k / 2.0k |
| 7 | ![]() claude-haiku-5.5 Anthropic | 87.6 | 94.1 | $6.16 | 32/34 | 4.3 | — | 56k / 1.0k |
| 8 | ![]() gemma-4-26b-a4b + hs-decide-1-flash Google · open weights | 86.7 | 85.3 | $1.84 | 29/34 | 2.6 | 15.5 | 9.0k / 217 |
| 9 | ![]() gpt-5.6-luna OpenAI | 86.7 | 94.1 | $7.46 | 32/34 | 4.3 | — | 37k / 458 |
| 10 | ![]() gemma-4-26b-a4b Google · open weights | 86.2 | 88.2 | $3.25 | 30/34 | 4.2 | — | 41k / 323 |
| 11 | ![]() minimax-m3.1-flash MiniMax · open weights | 86.1 | 91.2 | $5.37 | 31/34 | 5.0 | — | 64k / 2.2k |
| 12 | ![]() gemini-3.1-flash-lite Google | 85.9 | 94.1 | $9.01 | 32/34 | 4.2 | — | 41k / 348 |
| 13 | ![]() qwen3.8-27b Qwen · open weights | 85.8 | 94.1 | $9.29 | 32/34 | 4.4 | — | 50k / 2.0k |
| 14 | ![]() deepseek-v4.1-flash + hs-decide-1-flash DeepSeek · open weights | 85.6 | 85.3 | $2.34 | 29/34 | 2.6 | 15.9 | 9.5k / 1.6k |
| 15 | ![]() qwen3.6-35b-a3b Qwen · open weights | 85.5 | 91.2 | $6.14 | 31/34 | 4.6 | — | 51k / 3.1k |
| 16 | ![]() gemini-3.1-flash-lite + hs-decide-1-flash Google | 84.3 | 85.3 | $3.15 | 29/34 | 2.4 | 15 | 7.7k / 188 |
| 17 | ![]() gemini-3.8-flash + hs-decide-1-flash Google | 83.1 | 91.2 | $11 | 31/34 | 2.4 | 14.8 | 7.5k / 853 |
| 18 | ![]() nemotron-3.5-super-vl NVIDIA · open weights | 82.9 | 88.2 | $7.03 | 30/34 | 4.5 | — | 51k / 1.8k |
| 19 | ![]() gemini-3.8-flash Google | 79.5 | 94.1 | $40 | 32/34 | 4.4 | — | 45k / 1.4k |
| 20 | ![]() gpt-6.1-terra OpenAI | 78.7 | 94.1 | $47 | 32/34 | 4.3 | — | 46k / 306 |
| 21 | ![]() qwen3.8-flash Qwen · open weights | 78.0 | 79.4 | $5.21 | 27/34 | 4.9 | — | 64k / 2.7k |
| 22 | ![]() gpt-5.6-terra OpenAI | 76.6 | 94.1 | $77 | 32/34 | 4.4 | — | 39k / 425 |
| 23 | ![]() gpt-5.6-sol OpenAI | 72.8 | 94.1 | $183 | 31/51 | 4.4 | — | 44k / 360 |
Click a row for its categories and failed cases. Hover for how a column is computed.
Quality vs. cost across 16 models on Reflect
Top left is best value. Hover a point for details.
Cost per 1k cases includes the hs-decide-1-flash calls for the runs that use it.
Cases passed by category
Where each model wins and loses. The best in each category is outlined.
| Model | Precision | Update | Aggregation | No overreach | Source authority | Pages | Orchestration |
|---|---|---|---|---|---|---|---|
glm-5.3-flash | 6/6 | 4/4 | 3/4 | 4/4 | 6/8 | 4/4 | 4/4 |
glm-5.3-flash+ | 6/6 | 4/4 | 4/4 | 4/4 | 6/8 | 4/4 | 4/4 |
gpt-6-luna+ | 6/6 | 4/4 | 4/4 | 4/4 | 6/8 | 4/4 | 4/4 |
gpt-6-luna | 6/6 | 4/4 | 4/4 | 4/4 | 6/8 | 4/4 | 4/4 |
claude-haiku-5.5+ | 6/6 | 4/4 | 4/4 | 4/4 | 5/8 | 4/4 | 4/4 |
deepseek-v4.1-flash | 6/6 | 4/4 | 4/4 | 4/4 | 4/8 | 4/4 | 4/4 |
claude-haiku-5.5 | 6/6 | 4/4 | 4/4 | 4/4 | 6/8 | 4/4 | 4/4 |
gemma-4-26b-a4b+ | 6/6 | 4/4 | 3/4 | 4/4 | 5/8 | 3/4 | 4/4 |
gpt-5.6-luna | 6/6 | 4/4 | 4/4 | 4/4 | 6/8 | 4/4 | 4/4 |
gemma-4-26b-a4b | 6/6 | 4/4 | 2/4 | 4/4 | 6/8 | 4/4 | 4/4 |
minimax-m3.1-flash | 6/6 | 4/4 | 4/4 | 3/4 | 6/8 | 4/4 | 4/4 |
gemini-3.1-flash-lite | 6/6 | 4/4 | 4/4 | 4/4 | 6/8 | 4/4 | 4/4 |
qwen3.8-27b | 6/6 | 4/4 | 4/4 | 4/4 | 6/8 | 4/4 | 4/4 |
deepseek-v4.1-flash+ | 6/6 | 4/4 | 2/4 | 4/4 | 6/8 | 3/4 | 4/4 |
qwen3.6-35b-a3b | 6/6 | 4/4 | 4/4 | 4/4 | 6/8 | 3/4 | 4/4 |
gemini-3.1-flash-lite+ | 6/6 | 4/4 | 2/4 | 4/4 | 6/8 | 3/4 | 4/4 |
gemini-3.8-flash+ | 6/6 | 4/4 | 3/4 | 4/4 | 6/8 | 4/4 | 4/4 |
nemotron-3.5-super-vl | 6/6 | 4/4 | 4/4 | 3/4 | 5/8 | 4/4 | 4/4 |
gemini-3.8-flash | 6/6 | 4/4 | 4/4 | 4/4 | 6/8 | 4/4 | 4/4 |
gpt-6.1-terra | 6/6 | 4/4 | 4/4 | 4/4 | 6/8 | 4/4 | 4/4 |
qwen3.8-flash | 4/6 | 4/4 | 2/4 | 4/4 | 6/8 | 3/4 | 4/4 |
gpt-5.6-terra | 6/6 | 4/4 | 4/4 | 4/4 | 6/8 | 4/4 | 4/4 |
gpt-5.6-sol | 6/9 | 4/6 | 4/6 | 4/6 | 6/12 | 4/6 | 3/6 |
Categories
From the evals in this version. Click one for its cases.
Precisionthe answer differs from its near-misses by one attribute: a version, a service name, a number (3)
hq-release-prodhq-entity-porthq-numeric-outageUpdatewhat is true now, when the bank also holds what was true before (2)
hq-owner-latestsp-rotationAggregationa count across many facts; a rule together with its approved exception (2)
hq-count-billingsp-friday-deployNo overreacha year the bank does not cover; a rule true in one region only (2)
hq-scoped-2fahq-absent-2024Source authoritythe handbook against confident chatter; questions only one source answers (4)
sp-datastoresp-approvalssp-postmortemsp-upgrade-ownerPagesthe bank holds a page about the question, and the answer comes from it (2)
hq-release-prodhq-count-billingOrchestration (2)
fu-escalationfu-release
Also in this version, measured but not graded as cases: test_reflect_benchmark_coding_agent_decision_model







