Fast yes/no, pick-one and score answers with a probability for every option.
- Parameters: 0.8B
- Context: 65,536 tokens
- Languages: 8
- Serves on: vLLM
| Recall | Page refresh | Consolidation | Reflect | General decisions | Serving on CPU | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Overall mean | LoCoMo | LongMemEval | Wikipedia revisions | Refreshes skipped | New facts caught | Dialogue NLI | SQuAD 2.0 and HotpotQA | Decision Index | Judge 50 memories s | One decision ms | Throughput /s |
 hs-decide-1-flash Hindsight · 0.8B | 75.6 | 93.0 | 97.5 | 79.5 | 89.4 | 71.4 | 75.8 | 61.8 | 36.1 | 67.3 | 1,900.6 | 0.75 |
S StartLux-Decision 0.8B Instant StartLux · 0.8B | 74.5 | 91.5 | 100.0 | 68.5 | 45.0 | 87.7 | 83.0 | 66.8 | 53.8 | 1,781.4 | 2,014.6 | 0.029 |
v d3-lite vLLM Semantic Router · 0.8B | 73.4 | 83.0 | 100.0 | 72.0 | 47.2 | 92.3 | 88.0 | 67.8 | 36.9 | 1,106 | 1,096.6 | 0.046 |
M decider-0.8b Mapika · 0.8B | 66.6 7 of 8 measured | 94.0 | 97.5 | 56.0 | 78.3 | 37.7 | 51.7 | 50.7 | — | 1,583.9 | 1,955.9 | 0.031 |
Quality vs. speed across 3 models
Top left is best value. Hover a point for details.
Model, as its provider’s logoBest value: no other model is both faster and better
Speed from the Judge 50 memories column, on CPU. 1 model not measured on every column is left out.
Benchmarks
Scores are shown 0 to 100, higher is better; speed and input size as measured, with lower latency better. A shaded, outlined cell is the best in its column: blue when it is ours. Overall is the mean over the score columns.
- LoCoMoEvidence memory in the top 10 of up to 300 (%), 200 questions; 48.0 with no decision model
- LongMemEvalEvidence memory in the top 10 of up to 300 (%), 40 questions; 87.5 with no decision model
- Wikipedia revisionsRight call on whether a new memory changes the page (%), 400 cases
- Refreshes skippedCovered or off-topic memories skipped, no LLM call (%), 180 cases
- New facts caughtReal additions sent to the LLM refresh (%), 220 cases
- Dialogue NLIRight call on whether a new fact changes a stored one (%), 400 pairs
- SQuAD 2.0 and HotpotQARight answer to whether the evidence answers the question: not, partly, fully (%), 400 questions
- Decision IndexPublic suite, edition 0.3, 37 benchmarks; chance-corrected skill (x100)
- Judge 50 memoriesMedian seconds for one yes/no question on each of 50 memories, 2 CPU threads
- One decisionMedian ms for one yes/no question, 2 CPU threads
- ThroughputMemories judged per second, 2 CPU threads