Pick the LLM that keeps knowledge pages up to date as memories change.
- Embeddings: bge-small-en-v1.5
- Reranker: ms-marco-MiniLM-L-6-v2
- Hindsight: 1.2.0
- Judge: gemini-3.1-flash-lite
Results
20 cases in 7 categoriesEvery new memory is read by the LLM. Rows with + decision model also run one; open one to see what each model did and cost.
| # | Model | Cases | LLM calls | Decision calls | Tokens in / out | |||
|---|---|---|---|---|---|---|---|---|
| 1 | ![]() gpt-6-luna + hs-decide-1-flash OpenAI | 92.1 | 90.0 | $3.44 | 36/40 | 4.3 | 83.7 | 20k / 1.2k |
| 2 | ![]() gemini-3.1-flash-lite + hs-decide-1-flash Google | 89.9 | 90.0 | $5.78 | 36/40 | 4.0 | 82.5 | 17k / 871 |
| 3 | ![]() claude-haiku-5.5 + hs-decide-1-flash Anthropic | 83.9 | 82.5 | $6.80 | 33/40 | 4.4 | 83.7 | 32k / 3.4k |
| 4 | ![]() gemma-4-26b-a4b + hs-decide-1-flash Google · open weights | 83.4 | 77.5 | $3.42 | 31/40 | 4.2 | 82.7 | 22k / 824 |
| 5 | ![]() gpt-6-luna OpenAI | 83.4 | 82.5 | $7.69 | 33/40 | 8.9 | — | 77k / 2.2k |
| 6 | ![]() glm-5.3-flash + hs-decide-1-flash Z.ai · open weights | 83.0 | 77.5 | $3.79 | 31/40 | 4.3 | 83.7 | 22k / 8.8k |
| 7 | ![]() gemini-3.8-flash + hs-decide-1-flash Google | 82.6 | 90.0 | $31 | 36/40 | 3.9 | 82.6 | 16k / 4.1k |
| 8 | ![]() glm-5.3-flash Z.ai · open weights | 82.5 | 75.0 | $2.82 | 30/40 | 9.2 | — | 85k / 11k |
| 9 | ![]() gpt-5.6-luna OpenAI | 82.3 | 85.0 | $15 | 34/40 | 8.5 | — | 73k / 2.2k |
| 10 | ![]() claude-haiku-5.5 Anthropic | 82.2 | 85.0 | $15 | 34/40 | 9.3 | — | 125k / 5.4k |
| 11 | ![]() gemini-3.1-flash-lite Google | 81.6 | 85.0 | $17 | 34/40 | 7.7 | — | 64k / 1.5k |
| 12 | ![]() deepseek-v4.1-flash DeepSeek · open weights | 79.3 | 77.5 | $8.73 | 31/40 | 10.3 | — | 115k / 12k |
| 13 | ![]() deepseek-v4.1-flash + hs-decide-1-flash DeepSeek · open weights | 77.4 | 75.0 | $9.08 | 30/40 | 5.3 | 88.2 | 31k / 9.1k |
| 14 | ![]() gemma-4-26b-a4b Google · open weights | 77.3 | 72.5 | $6.18 | 29/40 | 8.1 | — | 75k / 1.5k |
| 15 | ![]() minimax-m3.1-flash MiniMax · open weights | 75.6 | 75.0 | $14 | 30/40 | 12.5 | — | 161k / 8.3k |
| 16 | ![]() gpt-6.1-terra OpenAI | 75.0 | 85.0 | $80 | 34/40 | 9.4 | — | 87k / 1.4k |
| 17 | ![]() nemotron-3.5-super-vl NVIDIA · open weights | 73.5 | 72.5 | $15 | 29/40 | 8.2 | — | 86k / 10k |
| 18 | ![]() gemini-3.8-flash Google | 73.4 | 82.5 | $77 | 33/40 | 8.2 | — | 74k / 4.9k |
| 19 | ![]() qwen3.6-35b-a3b Qwen · open weights | 72.7 | 70.0 | $12 | 28/40 | 8.3 | — | 94k / 7.0k |
| 20 | ![]() qwen3.8-27b Qwen · open weights | 71.8 | 75.0 | $33 | 30/40 | 9.6 | — | 107k / 12k |
| 21 | ![]() gpt-5.6-terra OpenAI | 67.1 | 77.5 | $146 | 31/40 | 7.9 | — | 72k / 1.9k |
| 22 | ![]() qwen3.8-flash Qwen · open weights | 64.5 | 57.5 | $11 | 23/40 | 8.8 | — | 124k / 9.1k |
| 23 | ![]() gpt-5.6-sol OpenAI | 59.7 | 70.0 | $240 | 27/60 | 6.3 | — | 60k / 974 |
Click a row for its categories and failed cases. Hover for how a column is computed.
Quality vs. cost across 16 models on Refresh pages
Top left is best value. Hover a point for details.
Cost per 1k cases includes the hs-decide-1-flash calls for the runs that use it.
Cases passed by category
Where each model wins and loses. The best in each category is outlined.
| Model | Precision | Update | Aggregation | No overreach | Stability | Consistency | Orchestration |
|---|---|---|---|---|---|---|---|
gpt-6-luna+ | 6/6 | 8/8 | 8/8 | 4/4 | 4/6 | 2/4 | 4/4 |
gemini-3.1-flash-lite+ | 5/6 | 8/8 | 7/8 | 4/4 | 6/6 | 2/4 | 4/4 |
claude-haiku-5.5+ | 6/6 | 8/8 | 6/8 | 4/4 | 4/6 | 1/4 | 4/4 |
gemma-4-26b-a4b+ | 5/6 | 8/8 | 4/8 | 4/4 | 6/6 | 0/4 | 4/4 |
gpt-6-luna | 6/6 | 7/8 | 8/8 | 4/4 | 4/6 | 0/4 | 4/4 |
glm-5.3-flash+ | 6/6 | 7/8 | 6/8 | 4/4 | 4/6 | 0/4 | 4/4 |
gemini-3.8-flash+ | 5/6 | 8/8 | 8/8 | 4/4 | 6/6 | 1/4 | 4/4 |
glm-5.3-flash | 5/6 | 7/8 | 6/8 | 3/4 | 4/6 | 1/4 | 4/4 |
gpt-5.6-luna | 6/6 | 6/8 | 8/8 | 4/4 | 4/6 | 2/4 | 4/4 |
claude-haiku-5.5 | 6/6 | 8/8 | 6/8 | 4/4 | 4/6 | 2/4 | 4/4 |
gemini-3.1-flash-lite | 6/6 | 6/8 | 5/8 | 4/4 | 6/6 | 3/4 | 4/4 |
deepseek-v4.1-flash | 5/6 | 8/8 | 5/8 | 4/4 | 4/6 | 1/4 | 4/4 |
deepseek-v4.1-flash+ | 6/6 | 8/8 | 4/8 | 4/4 | 4/6 | 0/4 | 4/4 |
gemma-4-26b-a4b | 6/6 | 6/8 | 4/8 | 4/4 | 6/6 | 0/4 | 3/4 |
minimax-m3.1-flash | 5/6 | 7/8 | 5/8 | 4/4 | 4/6 | 1/4 | 4/4 |
gpt-6.1-terra | 6/6 | 6/8 | 8/8 | 4/4 | 5/6 | 1/4 | 4/4 |
nemotron-3.5-super-vl | 6/6 | 6/8 | 6/8 | 4/4 | 4/6 | 0/4 | 3/4 |
gemini-3.8-flash | 6/6 | 6/8 | 5/8 | 4/4 | 6/6 | 2/4 | 4/4 |
qwen3.6-35b-a3b | 5/6 | 6/8 | 5/8 | 3/4 | 4/6 | 2/4 | 3/4 |
qwen3.8-27b | 6/6 | 6/8 | 5/8 | 4/4 | 4/6 | 2/4 | 3/4 |
gpt-5.6-terra | 6/6 | 8/8 | 7/8 | 4/4 | 4/6 | 0/4 | 2/4 |
qwen3.8-flash | 4/6 | 7/8 | 3/8 | 2/4 | 3/6 | 0/4 | 4/4 |
gpt-5.6-sol | 5/9 | 7/12 | 7/12 | 4/6 | 2/9 | 0/6 | 2/6 |
Categories
From the evals in this version. Click one for its cases.
Precisiona page built across waves keeps the asked attribute apart from its near-misses (3)
hq-release-prodhq-entity-porthq-numeric-outageUpdatea later wave changes the answer: a newer owner, a departed on-call, a renamed service, a corrected figure (4)
hq-owner-latestdu-oncall-departed-0du-renamed-service-0du-backup-correction-0Aggregationa count grows across waves; a list gains and loses members; a new risk joins the list; a table grows over four deltas (4)
hq-count-billingdu-enterprise-plan-0du-launch-risk-0du-incident-table-0No overreacha year the bank does not cover; a rule true in one region only (2)
hq-scoped-2fahq-absent-2024Stabilitya wave with nothing for the page leaves it intact; no page collapses across waves; a history re-recorded every wave is not re-added (3)
page-never-collapsesdu-quiet-wave-0du-rerecorded-history-0Consistencya changed value changes everywhere the page restates it (English, Chinese) (2)
pump-maintenance-en-0pump-maintenance-zh-0Orchestration (2)
fu-escalationfu-release
Also in this version, measured but not graded as cases: test_refresh_pages_benchmark_cost test_refresh_pages_benchmark_cost_all_pages







