See what Hindsight changes for a coding agent. Each agent fixes the same bugs with no memory and with Hindsight’s coding-agent plugin; fixed first try shows how often memory spares it a round of failing tests.
Results from: Agent Memory Benchmark
At commit: 28b644b
Dataset: SDE-Bench boltons, 61 bugs
Repeats: 3 per row, averaged
| Memory | ||||||
|---|---|---|---|---|---|---|
 opencode  google/gemini-3.5-flash |  Hindsightgemma-4-26b-a4b | 91.9 | 93.4 | $0.45 | 0.07 | 116 s |
 claude-code  claude-sonnet-5 | Hindsightgemma-4-26b-a4b | 84.0 | 80.3 | $0.37 | 0.20 | 83 s |
claude-code claude-sonnet-5 | Hindsightgemini-2.5-flash-lite | 76.7 | 69.4 | $0.34 * | 0.36 | 75 s |
 codex  gpt-5.4-mini | Hindsightgemini-2.5-flash-lite | 73.5 | 62.8 | $0.28 * | 0.47 | 49 s |
codex gpt-5.4-mini | Hindsightgemma-4-26b-a4b | 61.3 | 45.4 | $0.34 | 0.67 | 39 s |
claude-code claude-sonnet-5 | No memory | 54.4 | 38.8 | $0.45 | 0.85 | 85 s |
opencode google/gemini-3.5-flash | Hindsightgemini-2.5-flash-lite | 47.6 | 31.7 | $0.55 * | 0.80 | 163 s |
codex gpt-5.4-mini | No memory | 32.3 | 6.6 | $0.58 | 1.34 | 54 s |
opencode google/gemini-3.5-flash | No memory | 29.9 | 7.1 | $0.63 | 1.20 | 174 s |
Quality vs. cost across 3 agents on SDE-Bench
Top left is best value. Hover a point for details.
Agent with no memoryThe same agent with HindsightBest value: no other agent is both cheaper and better
The same agent with HindsightBest value: no other agent is both cheaper and betterHindsight runs on gemma-4-26b-a4b here. Cost per bug is the agent’s tokens plus Hindsight’s own memory bill, averaged over the repeats.