families who appeal a denied claim for their child usually win. almost nobody appeals, because the letters are too hard to write. that gap is the whole problem.
116/118
the strongest configuration finds 116 of 118 legal grounds on the current set. it also misuses legal authority in 16 of 30 cases.¹ the architecture carries the result, not the model — an 8B model on a mac mini with the full system beats a fine-tune of itself without it.
¹ misuse means a real provision, correctly formatted and
drawn from our own library, offered for a proposition it does not support.
it is not invented law. our automated check catches 46% of what the
independent grader finds, so the grader is doing the work here.
the answer key is written by one person — the
founder — so every number on this page measures agreement with him
until a health-law attorney reviews it.
do the traps make the model cite MORE of what it was warned about?
—the deepseek traps run scored higher and hallucinated more — cases with misused citations went from 10 of 30 to 12 of 30. naming a provision to warn against it may raise its salience. if that holds, the traps layer trades citation hygiene for recall, and we need to know the exchange rate before shipping it.
queued · exp-032 · after exp-031
can an ALJ's own reasoning replace golds we wrote ourselves?
—every recall number on this site measures agreement with one person. 212 new jersey administrative law decisions are now on disk; 59 are pediatric medicaid cases with reasoning, a holding, and at least three citations. a gold derived from a judge's written conclusion is third-party evidence. six of them also record an argument the judge rejected, which is the slice we cannot generate credibly ourselves.
queued · exp-026 · after exp-032
does the citation library hold at 200 cases instead of 50?
—50 cases proved the mechanism. 200 proves it isn't luck. requires building the expanded dataset first.
last run · exp-029 · finished 2026-09-09
the 8B finds 84 of 118 grounds; the 30B collapses to 51 and is a dead lane
—the frontier lane scores 115 of 118 on this set. the 8B running on a mac mini reaches 84, and it misuses legal authority in 25 of 30 cases — far more often than any hosted model. the 30B is worse, not better: it cannot hold the required output format, so its 51 is a formatting failure as much as a legal one. the honest read is that the local lane argues the law adequately and the paperwork poorly.
sonder-qwen3:8b / sonder-30b + citelib · how far do the local models get on the harder 30-case eval?
watching the machine workesc to close
opening the window…
live · every 30 seconds—
the bench
the same legal work, run on machines that differ by five orders of magnitude in size. the question is which capabilities require scale and which do not.
sonderan 8B model on a mac mini in the office. free to run and private by default. patient health information cannot leave a hospital network, so a system that only works on a frontier api is a system most providers cannot buy.
kimi k2a trillion-parameter model rented by the call, roughly 125,000× sonder's size.
claude fable 5a frontier model from anthropic. the strongest configuration on the bench, and the ceiling the cheaper lanes are measured against.
gpt-5.6openai's frontier model. it grades every run, and it is held fixed — changing the grader mid-experiment would invalidate every comparison to a baseline.
the ledger
4 d agofinding the law · anthropic_claude-fable-5-growth30-citefix + citation library98%found 115 of 118 grounds a specialist would raise. misused legal authority in 27 of 30 cases.
4 d agofinding the law · anthropic_claude-fable-5-growth30-citefix + citation library98%found 116 of 118 grounds a specialist would raise. misused legal authority in 16 of 30 cases.
5 d agofinding the law · deepseek_deepseek-v4-pro-growth30-growth30-traps2 + citation library91%found 107 of 118 grounds a specialist would raise. misused legal authority in 12 of 30 cases.
5 d agofinding the law · deepseek_deepseek-v4-pro-growth30-growth30-mid + citation library88%found 104 of 118 grounds a specialist would raise. misused legal authority in 10 of 30 cases.
13 d agofinding the law · sonder (8B, local) + citation library + revise loop71%found 84 of 118 grounds a specialist would raise. misused legal authority in 25 of 30 cases.
14 d agofinding the law · sonder (8B, local) + citation library71%found 84 of 118 grounds a specialist would raise. misused legal authority in 30 of 30 cases.
14 d agofinding the law · sonder-qwen3-8b-growth30 + citation library70%found 82 of 118 grounds a specialist would raise. misused legal authority in 30 of 30 cases.
40 d agofinding the law · anthropic_claude-fable-5-growth30-mech + citation library98%found 115 of 118 grounds a specialist would raise. misused legal authority in 6 of 30 cases.
41 d agofinding the law · anthropic_claude-fable-5-growth30-growth30 + citation library78%found 28 of 36 grounds a specialist would raise. misused legal authority in 7 of 9 cases.
42 d agofinding the law · anthropic_claude-fable-5-denial-cases-v0-teacher-v7 + citation library100%found 151 of 151 grounds a specialist would raise. misused legal authority in 24 of 50 cases.
42 d agofinding the law · anthropic_claude-fable-5-denial-cases-v0-teacher-v7 + citation library98%found 148 of 151 grounds a specialist would raise. misused legal authority in 19 of 50 cases.
42 d agofinding the law · sonder (8B, local) + citation library80%found 120 of 151 grounds a specialist would raise. misused legal authority in 47 of 50 cases.
42 d agofinding the law · mlx_mlx-community_Qwen3-8B-4bit-sample10-iter55-s10 + citation library70%found 21 of 30 grounds a specialist would raise. misused legal authority in 9 of 10 cases.
42 d agofinding the law · mlx_mlx-community_Qwen3-8B-4bit-held_out_cases-ft-holdout + citation library71%found 105 of 148 grounds a specialist would raise. misused legal authority in 43 of 49 cases.
42 d agofinding the law · mlx_mlx-community_Qwen3-8B-4bit-ft + citation library70%found 105 of 151 grounds a specialist would raise. misused legal authority in 46 of 50 cases.
43 d agofinding the law · anthropic_claude-fable-5-eval100v5 + citation library100%found 151 of 151 grounds a specialist would raise. misused legal authority in 18 of 50 cases.
43 d agofinding the law · anthropic_claude-fable-5-eval100v4 + citation library99%found 149 of 151 grounds a specialist would raise. misused legal authority in 24 of 50 cases.
43 d agofinding the law · anthropic_claude-fable-5-eval100v3 + citation library99%found 149 of 151 grounds a specialist would raise. misused legal authority in 25 of 50 cases.
43 d agofinding the law · sonder-qwen3-8b-prompts + citation library76%found 114 of 151 grounds a specialist would raise. misused legal authority in 44 of 50 cases.
43 d agofinding the law · anthropic_claude-fable-5-eval100v2 + citation library99%found 149 of 151 grounds a specialist would raise. misused legal authority in 19 of 50 cases.
43 d agofinding the law · anthropic_claude-fable-5-eval100 + citation library98%found 148 of 151 grounds a specialist would raise. misused legal authority in 13 of 50 cases.
43 d agofinding the law · sonder (8B, local) + citation library + revise loop82%found 124 of 151 grounds a specialist would raise. misused legal authority in 31 of 50 cases.
43 d agofinding the law · sonder (30B, local) + citation library74%found 111 of 151 grounds a specialist would raise. misused legal authority in 40 of 50 cases.
43 d agofinding the law · sonder (30B, local) + citation library72%found 109 of 151 grounds a specialist would raise. misused legal authority in 26 of 50 cases.
44 d agofinding the law · sonder (30B, local) + citation library + revise loop72%found 108 of 151 grounds a specialist would raise. misused legal authority in 35 of 50 cases.
44 d agofinding the law · claude fable + citation library98%found 148 of 151 grounds a specialist would raise. misused legal authority in 17 of 50 cases.
44 d agofinding the law · sonder (30B, local) + citation library + revise loop64%found 97 of 151 grounds a specialist would raise. misused legal authority in 36 of 50 cases.
44 d agofinding the law · kimi k2 (1T, API) + citation library + revise loop70%found 106 of 151 grounds a specialist would raise. misused legal authority in 14 of 50 cases.
44 d agofinding the law · claude fable + citation library + revise loop90%found 136 of 151 grounds a specialist would raise. misused legal authority in 11 of 50 cases.
44 d agofinding the law · gpt-5.5 + citation library + revise loop87%found 131 of 151 grounds a specialist would raise. misused legal authority in 8 of 50 cases.
44 d agofinding the law · kimi k2 (1T, API) + citation library66%found 99 of 151 grounds a specialist would raise. misused legal authority in 27 of 50 cases.
44 d agofinding the law · gpt-5.5 + citation library72%found 108 of 151 grounds a specialist would raise. misused legal authority in 4 of 50 cases.
44 d agofinding the law · gpt-5.6 + citation library90%found 136 of 151 grounds a specialist would raise. misused legal authority in 2 of 50 cases.
44 d agofinding the law · kimi k2.7 (API)69%found 104 of 151 grounds a specialist would raise. misused legal authority in 37 of 50 cases.
44 d agoreading denials · sonder (30B, local)94%given 50 denial letters, the model named the real reason for denial in 47. on the letters written to disguise it, 80%.
44 d agoreading denials · sonder (8B, local)96%given 50 denial letters, the model named the real reason for denial in 48. on the letters written to disguise it, 80%.
44 d agoreading denials · kimi k2.7 (API)94%given 50 denial letters, the model named the real reason for denial in 47. on the letters written to disguise it, 80%.
93 d agofinding the law · deepseek_deepseek-v4-pro87%found 131 of 151 grounds a specialist would raise. misused legal authority in 11 of 50 cases.
94 d agofinding the law · sonder (8B, local)76%found 114 of 151 grounds a specialist would raise. misused legal authority in 35 of 50 cases.
94 d agofinding the law · claude fable16%found 24 of 151 grounds a specialist would raise. misused legal authority in 0 of 50 cases.
94 d agofinding the law · gpt-5.593%found 141 of 151 grounds a specialist would raise. misused legal authority in 2 of 50 cases.
94 d agofinding the law · sonder (8B, local) + citation library + operational-grounds prompt70%found 106 of 151 grounds a specialist would raise. misused legal authority in 28 of 50 cases.
94 d agofinding the law · gpt-5.5 + citation library + operational-grounds prompt91%found 137 of 151 grounds a specialist would raise. misused legal authority in 1 of 50 cases.
95 d agodrafting appeals · sonder (8B, local)254 flaggedthe verifier read 50 complete appeal letters and flagged 254 unsupported claims. it does not catch a real provision cited for a proposition it does not hold.
95 d agodrafting appeals · kimi k2 (1T, API)281 flaggedthe verifier read 50 complete appeal letters and flagged 281 unsupported claims. it does not catch a real provision cited for a proposition it does not hold.
95 d agofinding the law · sonder (8B, local)63%found 95 of 151 grounds a specialist would raise. misused legal authority in 44 of 50 cases.
95 d agofinding the law · kimi k2 (1T, API)74%found 112 of 151 grounds a specialist would raise. misused legal authority in 47 of 50 cases.
95 d agoreading denials · kimi k2 (1T, API)98%given 50 denial letters, the model named the real reason for denial in 49. on the letters written to disguise it, 90%.