Pick the LLM that turns chats, documents and code into memories: complete, attributed, dated, and nothing invented.
Swapped on this board
LLM
Held fixed
- Embeddings: bge-small-en-v1.5
- Reranker: ms-marco-MiniLM-L-6-v2
- Hindsight: 1.2.0
- Judge: gemini-3.1-flash-lite
Results
42 cases in 12 categoriesDefault retain: one extraction call per chunk.
| # | Model | Cases | LLM calls | Tokens in / out | |||
|---|---|---|---|---|---|---|---|
| 1 |  glm-5.3-flash Z.ai · open weights | 95.0 | 92.9 | $0.21 | 40/42 | 2.3 | 7.9k / 647 |
| 2 |  minimax-m3.1-flash MiniMax · open weights | 94.6 | 98.8 | $0.60 | 41/42 | 1.5 | 4.8k / 641 |
| 3 |  gpt-6-luna OpenAI | 94.6 | 94.0 | $0.28 | 40/42 | 1.5 | 4.5k / 351 |
| 4 |  qwen3.8-flash Qwen · open weights | 94.1 | 97.6 | $0.55 | 42/42 | 1.5 | 5.2k / 832 |
| 5 |  deepseek-v4.1-flash DeepSeek · open weights | 92.6 | 94.0 | $0.44 | 38/42 | 1.7 | 5.4k / 662 |
| 6 |  claude-haiku-5.5 Anthropic | 90.2 | 96.4 | $1.13 | 41/42 | 1.5 | 6.9k / 875 |
| 7 | gpt-5.6-luna OpenAI | 89.6 | 92.9 | $0.73 | 39/42 | 1.5 | 4.5k / 443 |
| 8 | gpt-6.1-terra OpenAI | 87.3 | 97.6 | $2.65 | 41/42 | 1.5 | 4.5k / 365 |
| 9 |  gemini-3.1-flash-lite Google | 85.2 | 89.3 | $1.14 | 38/42 | 1.5 | 3.1k / 678 |
| 10 | gemini-3.8-flash Google | 85.1 | 95.2 | $2.98 | 40/42 | 1.5 | 3.1k / 623 |
| 11 | gemma-4-26b-a4b Google · open weights | 81.3 | 79.8 | $0.60 | 33/42 | 1.6 | 5.4k / 748 |
| 12 | gpt-5.6-terra OpenAI | 78.9 | 91.7 | $7.18 | 39/42 | 1.5 | 4.5k / 429 |
Click a row for its categories and failed cases. Hover for how a column is computed.
Quality vs. cost across 12 models on Retain
Top left is best value. Hover a point for details.
Model, as its provider’s logoBest value: no other model is both cheaper and better
Cases passed by category
Where each model wins and loses. The best in each category is outlined.
| Model | Selectivity | Completeness | Attribution | Temporal | Language | Fidelity | Mission | Mode | Entities | Entity labels | Inputs | No invention |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| glm-5.3-flash | 3/3 | 2/2 | 4/4 | 4/4 | 6/7 | 4/5 | 2/2 | 4/4 | 2/2 | 3/3 | 3/3 | 3/3 |
| minimax-m3.1-flash | 3/3 | 2/2 | 4/4 | 4/4 | 7/7 | 5/5 | 2/2 | 4/4 | 2/2 | 3/3 | 3/3 | 2/3 |
| gpt-6-luna | 3/3 | 2/2 | 4/4 | 4/4 | 5/7 | 5/5 | 2/2 | 4/4 | 2/2 | 3/3 | 3/3 | 3/3 |
| qwen3.8-flash | 3/3 | 2/2 | 4/4 | 4/4 | 7/7 | 5/5 | 2/2 | 4/4 | 2/2 | 3/3 | 3/3 | 3/3 |
| deepseek-v4.1-flash | 3/3 | 1/2 | 4/4 | 4/4 | 6/7 | 4/5 | 2/2 | 3/4 | 2/2 | 3/3 | 3/3 | 3/3 |
| claude-haiku-5.5 | 3/3 | 2/2 | 4/4 | 4/4 | 7/7 | 4/5 | 2/2 | 4/4 | 2/2 | 3/3 | 3/3 | 3/3 |
| gpt-5.6-luna | 3/3 | 2/2 | 3/4 | 4/4 | 6/7 | 5/5 | 2/2 | 4/4 | 2/2 | 3/3 | 3/3 | 2/3 |
| gpt-6.1-terra | 3/3 | 2/2 | 4/4 | 4/4 | 7/7 | 5/5 | 2/2 | 4/4 | 2/2 | 3/3 | 3/3 | 2/3 |
| gemini-3.1-flash-lite | 3/3 | 1/2 | 3/4 | 3/4 | 7/7 | 4/5 | 2/2 | 4/4 | 2/2 | 3/3 | 3/3 | 3/3 |
| gemini-3.8-flash | 3/3 | 2/2 | 4/4 | 4/4 | 7/7 | 5/5 | 2/2 | 3/4 | 2/2 | 3/3 | 3/3 | 2/3 |
| gemma-4-26b-a4b | 3/3 | 2/2 | 2/4 | 3/4 | 5/7 | 3/5 | 2/2 | 3/4 | 2/2 | 3/3 | 3/3 | 2/3 |
| gpt-5.6-terra | 3/3 | 2/2 | 3/4 | 4/4 | 6/7 | 5/5 | 2/2 | 4/4 | 2/2 | 3/3 | 3/3 | 2/3 |
Categories
From the evals in this version. Click one for its cases.
Selectivitychatter stores nothing; plans survive among small talk; a fact said three times is stored once (3)
chatter-onlysaid-three-timesdurable-among-noiseCompletenessexact figures, never rounded; sensory and emotional detail kept (2)
exact-figuressensory-and-emotionalAttribution"my roommate" becomes Tamsin; the user's rules are `world`, the agent's actions `experience`; a speaker named in context or metadata is credited (4)
coreference-rolesagent-actions-vs-user-rulescontext-names-the-speakermulti-speaker-attributionTemporalrelative dates resolved; a year spans the year; open-ended and undated facts; no event date, no invented date (4)
relative-dates-resolvedcoarse-dates-span-the-periodopen-ended-and-undatedno-event-dateLanguageGerman, Chinese, Spanish keep their language and script; English, Italian, Japanese inputs retained 6× never switch (7)
german-with-english-termschinese-scriptspanish-conversationen-coding-sessionen-git-commitit-coding-sessionja-personal-noteFidelityhedges stay hedged, past states stay past, a correction wins, the prompt's examples never leak, instructions inside content are not obeyed (5)
hedges-stay-hedgednegation-and-past-statein-text-correctionexamples-do-not-leakinstructions-in-contentMissionthe retain mission filters what is kept and outranks the generic selectivity rules (2)
mission-filtersmission-overrides-selectivityModecustom instructions, verbose, verbatim, a strategy chosen per item (4)
custom-instructionsverbose-keeps-detailverbatim-preserves-textstrategy-per-itemEntitiesnamed entities stored; one person across documents is one entity (2)
entities-namedentities-across-documentsEntity labelsvalue and multi-value labels, free-text and map labels, labels-only mode (3)
labels-value-and-multilabels-free-text-and-maplabels-only-modeInputsdocument metadata read, JSON conversations, a fact in the last chunk of a long document (3)
metadata-readjson-conversationlate-fact-in-long-documentNo inventiona date, owner or place stated for a neighbouring subject is not borrowed (3)
date-borrowed-from-nearby-subjectowner-borrowed-from-nearby-subjectplace-borrowed-from-nearby-subject