A vendor-neutral, reproducible benchmark of United States legal-answer quality. A committed question set, a scoring script anyone can run, and metric definitions written before the run. Snapshot: July 2026.

This is a tool, not a testimonial. The benchmark (the questions and the scorer) is separate from any one system's results. Any legal-AI product runs the same questions through the same code and gets a comparable number, and a reader reproduces that number by rerunning the script. The maintainers build a legal-AI product and submit to this benchmark like anyone else, with the same disclosure and the same re-scorable answers (see Disclosure).

Most legal-AI accuracy claims are either a slogan with no number or a vendor benchmark whose questions are never released. The independent Stanford RegLab study found leading legal-research tools still describe the law incorrectly or miscite it 17 to 33 percent of the time, despite marketing that the problem was solved. This repository is the opposite: the exact questions, the exact scoring code, and metrics you can reproduce without trusting anyone.

The headline metrics need no model, no API key, and no trust. Clone the repo, run the script on a committed answers file, and you get the same number every time:

• must_include_accuracy the correct key fact (a statute section, a dollar figure, a case name) is present.
• citation_support_rate every [N] marker points to a source that actually supports the adjacent claim (scoring/deterministic_citation.py).
• citation_in_range_rate every [N] resolves to a real source, never a fabricated index.
• must_not_clean_rate no wrong-jurisdiction authority appears.
• right_authority_retrieved the expected primary authority is present in the cited sources.

These are the ranked leaderboard columns, because "rerun it and get what we got" is the whole point.

The subjective dimensions (faithfulness, coverage) come from an optional LLM judge. An LLM judge is noisy and gameable, so here it is anchored, sampled, and its prompt is published, and it only counts when its agreement with human graders is reported (scoring/human_validation.py). Deterministic first, judge second, human-validated.

On US law that changed in the last two years, a retrieval-grounded legal engine answered correctly 93 percent of the time versus 33 percent for the same model with no sources, and cited the governing authority 97 percent of the time against the model's zero. Every hard-set answer is verified against a primary source twice, the hard set is screened against a plain model, keeping only questions it gets wrong from memory, and the scorer is public. You do not have to trust the number: the answers, including the retrieved source text, are committed, so you can re-score them yourself and get it.

On famous, settled law the grounded engine matches a strong general model and cites a source the model cannot. On law that changed recently, the model collapses and grounding wins: the grounded engine answered the current law correctly on 93 percent of the hard tranche versus the ungrounded model's 33 percent, and retrieved the correct governing authority 97 percent of the time versus the model's zero. The hard tranche was verified against primary sources by two independent passes and screened against a plain model, keeping only questions it gets wrong from memory, so the win is not a memorization artifact. Full numbers, including the cases where the submitted system fell short, are in RESULTS.md.

A plain-language walkthrough of the result and the method is in the companion writeup, Grounding vs guessing: on US law that changed this year, a general model is wrong most of the time.

data/
  us_chat_golden_hard.jsonl     29 questions on recent/changed or obscure US law (the spine:
                                verified against primary sources, screened so a plain model gets them wrong from memory)
  us_chat_golden_control.jsonl  25 famous, settled black-letter questions (the control)
  us_chat_adversarial.jsonl     8 unanswerable / false-premise traps (judge-scored, directional)
scoring/
  score.py                     the scorer, standard library only
  make_charts.py               regenerates the SVG charts from a results file (no dependency)
  deterministic_citation.py    the deterministic citation-support metric
  human_validation.py          worksheet + judge-vs-human agreement
  judge_prompt.md              the exact optional-judge rubric
submissions/                   raw answers (both arms, both tranches) so you can re-score any submitted run
results/                       per-item results (both arms) + the chart inputs
charts/                        the rendered SVG charts
REPRODUCE.md                   exact commands to reproduce the charts and the numbers
GUIDE.md                       plain-language walkthrough of every term
SUBMISSION.md                  how any system runs and submits
LEADERBOARD.md                 results, ranked by the deterministic metrics
CONTRIBUTING.md                correcting an item, submitting a result
RESULTS.md                     metric definitions and methodology

Each dataset line carries must_include (distinctive substrings a correct answer should contain), must_not_include (wrong-jurisdiction tells), and expected_authorities (the primary authority a strong answer should rest on).

Produce an answers file (one JSON per line: {id, answer, sources}), then:

python scoring/score.py --golden data/us_chat_golden_hard.jsonl --answers your_answers.jsonl

New to benchmarks, or want every term explained in plain language? Start with GUIDE.md. The full submission format, the fairness rules, and how to appear on the leaderboard are in SUBMISSION.md. Want to reproduce the committed numbers and charts exactly? See REPRODUCE.md.

The maintainers build Vaquill, a US legal-AI product, so they have a stake in the result. The benchmark is built so that stake does not matter: the questions and scorer are here, the headline metrics are reproducible by anyone, RESULTS.md reports where the submitted system falls short, and its results are submitted under submissions/ with re-scorable answers, exactly like any other system. This is a self-run benchmark, not a third-party audit. The strongest external check is to run your own system through it, or to compare against an independent grader such as Vals AI or the open LegalBench-RAG.

A single wrong "correct answer" discredits a benchmark, so corrections are the most valuable contribution. If you believe an item's expected answer or authority is wrong, open an issue with the primary source. See CONTRIBUTING.md.

Data (the question sets, results, and submitted answers) is licensed under CC BY 4.0: reuse it freely, including commercially, with attribution. Code (the scorer and scripts) is licensed under the MIT License.

GitHub shows a "Cite this repository" button generated from CITATION.cff. Or cite it directly:

Open Legal-Answer Quality Benchmark (2026-07 snapshot). CC BY 4.0.
https://github.com/Vaquill-AI/open-legal-answer-benchmark