The Citation Record

Score an item

One sample item, in the interface the panel uses. It takes about ninety seconds. Every mechanical check on it passes: the case exists, the citation is accurate to volume and page, and the quoted words appear in the opinion. Scoring it is the point, so it is worth doing before reading on.

Legal AI systems produce citations. Some of those citations are to cases that do not exist. Others are to real cases with the wrong reporter or page, real cases that say something other than what the system claims, real cases that have been overruled, or real cases from a jurisdiction that does not bind the court in question.

These are six different failures with six different professional consequences, and the standard verification a lawyer performs — confirming the case exists — catches only the first. Every vendor claim we surveyed reports existence, or reports nothing measurable at all. The Citation Record scores all six separately, against a ground-truth corpus of public-domain primary law, owned by none of the systems under test, and publishes the false-positive rate alongside the catch rate.

How it works

The coverage check behind this benchmark once reported Mata v. Avianca as not found — the most cited real case in this field. It had read a sparsely ingested volume as fully indexed. CourtListener holds two opinions in 678 F. Supp. 3d, at pages 1369 and 1371; the cited page 443 is nowhere near them. Scored, that is a real case recorded as a fabrication.

The second failure was found by trying to reproduce a figure this project had already published, and failing. A probe sampled four volumes of Cal. App. 5th and three returned nothing. All three sit past volume 39, where CourtListener’s holdings of that reporter stop. Those zeros describe the sample, not the corpus.

Counted per cluster over a named bulk generation, 14,187 clusters carry a Cal. Rptr. 3d citation and no Cal. App. 5th citation, against 2,614 carrying both. A case absent from one reporter is usually held under the other. Read per reporter, the same data says something false.

Neither was repaired quietly. Both are listed in Known Weaknesses, by name, with the records behind them. The methodology is published before the results it produces, and the tooling and the verification records are citable, so every failure on that page can be checked against the repositories rather than taken on this page’s word.

Citation existence and accuracy resolve mechanically against CourtListener. Misattributed holdings, overruled authority, and jurisdictional weight require attorney judgment and receive it.

No AI system scores any item. The scoring rubric is published in full, and the methodology and prompt protocol are published before any results are. The query set is pre-registered rather than published: the queries and their expected answers are fixed and hashed before any model is called, and that hash is written to an append-only record before the first query. The cases an edition draws on are not held back, and cannot be: ground truth is established by resolving each citation against CourtListener, and every lookup is written to a public verification journal, because a verification that names nothing verifies nothing. What is held back is the rest — the queries themselves, the negated holdings, and the recorded answers — none of which is published until the edition ships, after which they are rotated for the next. Test items are derived from what the models return, so they are published with the edition rather than before it. What this does and does not protect is stated in Known Weaknesses. Each edition is free, carries a permanent identifier, and states the measurement window it describes.

Disclosures

No vendor pays anything

No measured vendor has paid, or will pay, any amount in any form: no participation fees, sponsorship, licensing, consulting, equity, referral commissions, or complimentary access beyond publicly available free tiers. This is a permanent commitment, not a policy for one edition.

The publisher sells work on this subject, and intends to sell a service built on this benchmark

The publisher sells a book on this subject and intends to offer an attestation service built on this benchmark’s methodology. A benchmark that is trusted and widely cited raises the value of both. Citation Firewall, a verification tool also built by the publisher, is not among the systems this benchmark tests. These interests are disclosed rather than resolved: nothing in the structure of the benchmark prevents its findings from serving the publisher’s other work.

The ground-truth provider integrates with products built on systems under test, and the benchmark pays for access

As of 21 September 2026, Free Law Project publishes an MCP server that connects CourtListener to AI assistants over an open standard. It reaches three products built on systems this benchmark measures: the Claude app, announced 12 May 2026; ChatGPT Enterprise, whose CourtListener plugin launched 17 September 2026; and Google Cloud’s Gemini Enterprise for Legal, announced 25 August 2026, which ships a citation verification capability.

It does not reach the configuration under test. Edition One measures Claude, GPT-5 and Gemini as raw models through their APIs, with retrieval disabled and the exact request setting that disables it recorded for each: for all three, the absence of any tools field, so no MCP server is reachable by the model at all. That is checkable against the stored request rather than taken on this page’s word. The relationship is disclosed because it exists, not because it touches a measurement.

The publisher has held a paid Tier 1 CourtListener membership, $10 a month at published rates, since 7 September 2026. This is disclosed rather than resolved: the corpus is public-domain primary law, one server serves every client on the same terms, and the alternatives — Shepard’s, KeyCite — are owned outright by systems under test.

The date is load-bearing, and the link goes to an archived copy rather than the live post. Which products reach this corpus is a fact about a third party’s integrations, it changes without notice, and a present-tense sentence citing a page that can be edited would hide its own staleness twice over. Read the list as what was true on the day stated and not as a standing description.

No non-disclosure agreements

The publisher has signed no NDA with any measured vendor and will not. Where a conflict cannot be disclosed, the affected system is withdrawn from testing and the withdrawal is stated.

Vendors see results before publication, and cannot change them

Each measured vendor receives its results 21 days before publication. Responses are printed unedited. Corrections are adopted where warranted; disputes are left standing on the record.

Status

First edition: 2026.Q4
Ground truth: CourtListener bulk data, generation stated per edition
Licence: Creative Commons Attribution 4.0
Tooling: github.com/CitationRecord
Scoring rubric: The six classes, as scorers apply them

Taking part

Edition One measures three systems: Claude, GPT-5 and Gemini, as raw models through their APIs. No participant needs a subscription to anything, and no participant accepts a vendor’s terms of use. The research and drafting products built on these models are not measured this edition; testing those waits on a seated panel and on counsel.

Class 4, superseded authority, is likely to be unscoreable from this corpus. CourtListener carries no treatment signal, so whether a case is still good law is attorney judgment rather than a field to read. At the scale an edition needs, it will probably be deferred rather than reported thin.

The panel is being assembled. Attorneys who do litigation work are welcome to write, as are law librarians willing to review the methodology before it is fixed. Scoring calls for judgment about holdings, subsequent history and jurisdictional weight, not for access to any vendor’s system. The commitment is roughly ninety minutes per quarter. Contributors are credited by name; none are paid.

jb@citationrecord.org