Pick the LLM that keeps knowledge pages up to date as memories change.
Swapped on this board
LLM
Held fixed
- Embeddings: bge-small-en-v1.5
- Reranker: ms-marco-MiniLM-L-6-v2
- Hindsight: 1.2.0
- Judge: gemini-3.1-flash-lite
Results
20 cases in 7 categoriesEvery new memory is read by the LLM. Rows with + decision model also run one; open one to see what each model did and cost.
| # | Model | Cases | LLM calls | Decision calls | Tokens in / out | |||
|---|---|---|---|---|---|---|---|---|
| 1 |  gpt-6-luna + hs-decide-1-flash OpenAI | 90.0 | 90.0 | $3.16 | 18/20 | 4.3 | 83.7 | 20k / 674 |
| 2 |  gemini-3.1-flash-lite + hs-decide-1-flash Google | 87.4 | 90.0 | $5.78 | 17/20 | 4.0 | 82.5 | 17k / 871 |
| 3 | gemini-3.8-flash + hs-decide-1-flash Google | 82.5 | 90.0 | $18 | 19/20 | 3.9 | 82.6 | 16k / 887 |
| 4 |  glm-5.3-flash Z.ai · open weights | 82.5 | 75.0 | $1.59 | 14/20 | 9.2 | — | 85k / 2.9k |
| 5 |  claude-haiku-5.5 + hs-decide-1-flash Anthropic | 82.1 | 82.5 | $5.90 | 17/20 | 4.4 | 83.7 | 32k / 1.6k |
| 6 | glm-5.3-flash + hs-decide-1-flash Z.ai · open weights | 81.3 | 77.5 | $3.13 | 16/20 | 4.3 | 83.7 | 22k / 1.8k |
| 7 | gpt-6-luna OpenAI | 81.1 | 82.5 | $7.33 | 16/20 | 8.9 | — | 77k / 1.5k |
| 8 | gemma-4-26b-a4b + hs-decide-1-flash Google · open weights | 80.9 | 77.5 | $3.42 | 16/20 | 4.2 | 82.7 | 22k / 824 |
| 9 | claude-haiku-5.5 Anthropic | 80.0 | 85.0 | $14 | 17/20 | 9.3 | — | 125k / 3.2k |
| 10 | gpt-5.6-luna OpenAI | 80.0 | 85.0 | $14 | 17/20 | 8.5 | — | 73k / 1.6k |
| 11 |  deepseek-v4.1-flash DeepSeek · open weights | 79.6 | 77.5 | $4.61 | 16/20 | 10.3 | — | 115k / 4.1k |
| 12 | gemini-3.1-flash-lite Google | 79.1 | 85.0 | $17 | 17/20 | 7.7 | — | 64k / 1.5k |
| 13 | deepseek-v4.1-flash + hs-decide-1-flash DeepSeek · open weights | 76.7 | 75.0 | $6.05 | 15/20 | 5.3 | 88.2 | 31k / 3.1k |
| 14 | gemma-4-26b-a4b Google · open weights | 74.9 | 72.5 | $6.18 | 13/20 | 8.1 | — | 75k / 1.5k |
| 15 |  minimax-m3.1-flash MiniMax · open weights | 73.6 | 75.0 | $12 | 15/20 | 12.5 | — | 161k / 5.9k |
| 16 | gpt-6.1-terra OpenAI | 72.5 | 85.0 | $79 | 18/20 | 9.4 | — | 87k / 1.2k |
| 17 | gemini-3.8-flash Google | 71.6 | 82.5 | $65 | 16/20 | 8.2 | — | 74k / 1.9k |
| 18 | gpt-5.6-terra OpenAI | 64.8 | 77.5 | $139 | 17/20 | 7.9 | — | 72k / 1.3k |
| 19 |  qwen3.8-flash Qwen · open weights | 63.5 | 57.5 | $7.54 | 10/20 | 8.8 | — | 124k / 2.8k |
Click a row for its categories and failed cases. Hover for how a column is computed.
Quality vs. cost across 12 models on Refresh pages
Top left is best value. Hover a point for details.
Model, as its provider’s logo
The same model with hs-decide-1-flash, at the arrow's headBest value: no other model is both cheaper and better
The same model with hs-decide-1-flash, at the arrow's headBest value: no other model is both cheaper and betterCost per 1k cases includes the hs-decide-1-flash calls for the runs that use it.
Cases passed by category
Where each model wins and loses. The best in each category is outlined.
| Model | Precision | Update | Aggregation | No overreach | Stability | Consistency | Orchestration |
|---|---|---|---|---|---|---|---|
| gpt-6-luna+hs-decide-1-flash | 3/3 | 4/4 | 4/4 | 2/2 | 2/3 | 1/2 | 2/2 |
| gemini-3.1-flash-lite+hs-decide-1-flash | 2/3 | 4/4 | 3/4 | 2/2 | 3/3 | 1/2 | 2/2 |
| gemini-3.8-flash+hs-decide-1-flash | 3/3 | 4/4 | 4/4 | 2/2 | 3/3 | 1/2 | 2/2 |
| glm-5.3-flash | 2/3 | 3/4 | 3/4 | 2/2 | 2/3 | 0/2 | 2/2 |
| claude-haiku-5.5+hs-decide-1-flash | 3/3 | 4/4 | 3/4 | 2/2 | 2/3 | 1/2 | 2/2 |
| glm-5.3-flash+hs-decide-1-flash | 3/3 | 4/4 | 3/4 | 2/2 | 2/3 | 0/2 | 2/2 |
| gpt-6-luna | 3/3 | 3/4 | 4/4 | 2/2 | 2/3 | 0/2 | 2/2 |
| gemma-4-26b-a4b+hs-decide-1-flash | 3/3 | 4/4 | 2/4 | 2/2 | 3/3 | 0/2 | 2/2 |
| claude-haiku-5.5 | 3/3 | 4/4 | 3/4 | 2/2 | 2/3 | 1/2 | 2/2 |
| gpt-5.6-luna | 3/3 | 3/4 | 4/4 | 2/2 | 2/3 | 1/2 | 2/2 |
| deepseek-v4.1-flash | 3/3 | 4/4 | 2/4 | 2/2 | 2/3 | 1/2 | 2/2 |
| gemini-3.1-flash-lite | 3/3 | 3/4 | 2/4 | 2/2 | 3/3 | 2/2 | 2/2 |
| deepseek-v4.1-flash+hs-decide-1-flash | 3/3 | 4/4 | 2/4 | 2/2 | 2/3 | 0/2 | 2/2 |
| gemma-4-26b-a4b | 3/3 | 3/4 | 1/4 | 2/2 | 3/3 | 0/2 | 1/2 |
| minimax-m3.1-flash | 3/3 | 4/4 | 2/4 | 2/2 | 2/3 | 0/2 | 2/2 |
| gpt-6.1-terra | 3/3 | 3/4 | 4/4 | 2/2 | 3/3 | 1/2 | 2/2 |
| gemini-3.8-flash | 3/3 | 3/4 | 3/4 | 2/2 | 3/3 | 0/2 | 2/2 |
| gpt-5.6-terra | 3/3 | 4/4 | 4/4 | 2/2 | 2/3 | 0/2 | 2/2 |
| qwen3.8-flash | 2/3 | 3/4 | 1/4 | 1/2 | 1/3 | 0/2 | 2/2 |
Categories
From the evals in this version. Click one for its cases.
Precisiona page built across waves keeps the asked attribute apart from its near-misses (3)
hq-release-prodhq-entity-porthq-numeric-outageUpdatea later wave changes the answer: a newer owner, a departed on-call, a renamed service, a corrected figure (4)
hq-owner-latestdu-oncall-departed-0du-renamed-service-0du-backup-correction-0Aggregationa count grows across waves; a list gains and loses members; a new risk joins the list; a table grows over four deltas (4)
hq-count-billingdu-enterprise-plan-0du-launch-risk-0du-incident-table-0No overreacha year the bank does not cover; a rule true in one region only (2)
hq-scoped-2fahq-absent-2024Stabilitya wave with nothing for the page leaves it intact; no page collapses across waves; a history re-recorded every wave is not re-added (3)
page-never-collapsesdu-quiet-wave-0du-rerecorded-history-0Consistencya changed value changes everywhere the page restates it (English, Chinese) (2)
pump-maintenance-en-0pump-maintenance-zh-0Orchestration (2)
fu-escalationfu-release
Also in this version, measured but not graded as cases: test_refresh_pages_benchmark_cost test_refresh_pages_benchmark_cost_all_pages