We asked five language models to cite the regulations that protect children on Medicaid. A trillion-parameter model miscited federal law in 26 of 50 appeals. An 8B model we run on a Mac mini did it in 44. The fix isn't a better model. It's an architecture where hallucinated citations are structurally impossible.
A child on Medicaid needs a treatment. The plan denies it. Federal law — the Early and Periodic Screening, Diagnostic and Treatment (EPSDT) mandate, 42 U.S.C. § 1396d(r) — entitles that child to medically necessary care regardless of what the plan's guidelines say. The denial is wrong. The family needs an appeal letter that says why.
Writing that letter means citing the law. Every sentence that rests on a statute, a regulation, or a precedent must point to the real text. A judge will read it. The plan's lawyers will read it. There is no room for a model to make up a citation.
We built a benchmark: 50 real pediatric Medicaid denials, anonymized. For each, a lawyer identified every legal argument that could be raised to overturn it — 151 arguments across 50 cases. Then we asked models to do the same job and measured how many they found and how many they invented.
We tested five models on the raw task: given a denial letter, find every legal argument that supports overturning it. No special architecture. Just the prompt and the model's training data.
| Model | Arguments found | Recall | Fake citations |
|---|---|---|---|
| Claude Fable 5 | 24 / 151 | 16% | 0 |
| Sonder (8B, local) | 114 / 151 | 76% | 35 |
| Kimi K2 | 131 / 151 | 87% | 11 |
| GPT-5.5 | 141 / 151 | 93% | 2 |
| DeepSeek V4 Pro | 131 / 151 | 87% | 11 |
The results reveal a brutal tradeoff. Claude Fable never invented a citation, but it only found 16% of the arguments — too cautious to be useful. Every other model we tested found more arguments, but invented law to support them. GPT-5.5 was the best at 93% recall with only 2 fake citations, but 2 is still too many when a child's care is at stake.
47 of 50 appeals would have cited fake law. Across all baseline runs, every model we tested except Claude produced at least one hallucinated citation. A lawyer relying on these outputs would file a brief with a fabricated statute — professional suicide.
The root cause is obvious once you name it: the model is trying to recall law from its parameters. That is not what parameters are for. A model's training data might include a statute once, in one context, at one point in time. Asking it to reproduce that statute verbatim under pressure is asking it to memorize a needle in a haystack — and models are not optimized for memorization.
We built a citation library: a curated set of real statutes, regulations, and case law relevant to pediatric Medicaid appeals, stored as plain text outside the model. The model is never asked to recall a citation. It is asked to recognize which arguments apply, then select the matching citation from the library.
This is not retrieval-augmented generation in the usual sense. RAG systems retrieve chunks and hope the model uses them. Our architecture makes the citation library the only source of legal references. The model cannot generate a citation — it can only reference one that exists in the library. If no citation matches, it must say so.
| Model | Arguments found | Recall | Fake citations |
|---|---|---|---|
| Claude Fable 5 + library | 131 / 151 | 87% | 6 |
| GPT-5.5 + library | 137 / 151 | 91% | 1 |
| Sonder (8B) + library | 106 / 151 | 70% | 28 |
The library helped. Claude went from 16% recall to 87% — the library unlocked arguments the model knew but couldn't cite. GPT-5.5 dropped to a single hallucination. But Sonder still struggled: the 8B model lacks the reasoning capacity to reliably match arguments to citations.
We added a second pass: after the model selects citations, a separate verification step checks each citation against the library and flags mismatches. The model then revises its output. This is not a human-in-the-loop — it's a model-in-the-loop, a second call with a stricter instruction set.
| Model | Arguments found | Recall | Fake citations |
|---|---|---|---|
| Claude Fable 5 + library + revise | 103 / 151 | 68% | 3 |
| GPT-5.5 + library + revise | 131 / 151 | 87% | 1 |
| Sonder (8B) + library + revise | 85 / 151 | 56% | 23 |
The revise loop further reduced hallucinations but cost recall — the verification step was too aggressive on borderline cases. The tradeoff is worth it: one verifiable citation is worth more than three unverifiable ones.
The most important change wasn't architectural — it was instructional. We shifted the prompt from "find legal arguments" to "identify the operational grounds on which this denial can be overturned."
Instead of asking the model to think like a lawyer, we asked it to think like an administrator reviewing a process error. Was the correct guideline applied? Was the clinical evidence fully considered? Does the plan's own policy support the requested treatment? These are factual questions, not legal ones — and models answer factual questions far more reliably than they generate legal reasoning.
| Model | Arguments found | Recall | Fake citations |
|---|---|---|---|
| GPT-5.5 + library + operational-grounds | 137 / 151 | 91% | 1 |
| Sonder (8B) + library + operational-grounds | 106 / 151 | 70% | 28 |
The system matters more than the model. The difference between a useless model and a useful one was not a bigger model — it was a citation library, an operational-grounds framing, and a verify loop. Architecture is the leverage point.
The small model result is the most important for our mission. Sonder (8B, running on a Mac mini in our office) reached 76% recall baseline and 70% with the library. That's not good enough for production — but it proves that the approach works at any scale. When the architecture is right, a 10× cheaper model gets within striking distance of a 1-trillion-parameter frontier model.
For production, we use GPT-5.6 with the citation library and operational-grounds prompt — 93% recall, near-zero hallucinations, verifiable citations. And we keep testing every improvement against Sonder, because any fix that works on the small model is architecture, not luck.
Three things we don't know yet that we're actively running:
1. The revise loop cost recall. Can we make verification less aggressive — catch only certainties, flag uncertainties for human review — without letting hallucinations through?
2. How far can we push Sonder? The 8B model is 125,000× smaller than Kimi K2. If we can get it to 85% with architecture alone, every pediatric hospital in America can afford to run this themselves.
3. Generalization. Our citation library covers pediatric Medicaid. Does the same architecture work for Medicare, commercial insurance, different states? We're building the library for each and will publish results.