Known Weaknesses

A sparse volume read as full coverage

sparse-volume-read-as-full · found 3 September 2026 · fixed

The defect. The coverage check treated any volume CourtListener held at all as fully indexed. A citation to a page the corpus had never ingested therefore resolved as not found rather than not covered. Scored, that is a false positive for fabrication recorded against a real case.

Evidence. It reported Mata v. Avianca, 678 F. Supp. 3d 443, as not found. That is the most cited real case in this field, and the reason the field exists. CourtListener holds exactly two opinions in volume 678 of F. Supp. 3d, at pages 1369 and 1371, both United States Court of International Trade decisions from 2024. Page 443 is nowhere near the ingested region.

Lookup	Retrieved	Outcome
678 F. Supp. 3d 443	2026-09-08T05:11:46Z	not_covered, volume_only_partially_held
678 F. Supp. 3d 443	2026-09-08T18:47:29Z	not_covered, volume_only_partially_held
678 F. Supp. 3d 1369	2026-09-10T03:14:33Z	cluster 9467618, court cit
678 F. Supp. 3d 1371	2026-09-10T03:15:08Z	cluster 9468905, court cit

Records in CitationRecord/resolutions at 20f3823. Both surviving lookups of page 443 already show the corrected verdict, so the misclassification predates the journal and is attested by the commit that repaired it rather than by a record of it happening. The court was established later and from the API, not from the original probe.

What fixed it. CitationRecord/tooling at 9276206, which states it as rejecting a sparse volume as evidence of absence. The resolver now probes what the volume actually holds and returns not_covered with the coverage counts attached.

Residual, accepted. The resolver still cannot separate a page CourtListener never ingested from a page that was invented. Nothing available from CourtListener can. Such a citation goes to the unscorable bucket with its coverage counts, and an attorney decides. That is the direction the methodology requires: a coverage gap scored as a fabrication is worse than a fabrication left unscored, because it inflates error rates against systems drawing on broader corpora than the ground-truth source.

A phrase query matched reporters by prefix

reporter-prefix-match · found 3 September 2026 · fixed

The defect. Coverage counts came from a phrase query against citation text. A phrase query is a prefix match, so citation:("347 U.S.") also matched 347 U.S. App. D.C. Every reporter whose name prefixes another was counted as denser than it is.

Evidence. Volume 347 of U.S. returned 734 under the phrase query against 691 under the structured filter, the difference being the 43 opinions in volume 347 of U.S. App. D.C.

What it caused. No published figure, but the same false-positive direction as the entry above. That count is what decides not found against not covered. An inflated count makes a volume look well held, which pushes a citation the corpus does not carry toward being read as fabricated rather than as uncovered. Two independent defects tilting the instrument the same way is the part worth remembering.

What fixed it. The same commit, 9276206. Counts now come from the structured citation filter keyed on the reporter field, which matches exactly, and both the resolver and the census use that filter so the two components measure the same thing the same way. Kept honest by two named tests, test_reporter_prefix_does_not_inflate_a_volume and test_a_prefix_reporter_stays_distinct.

Residual. None known for this query. The general form has no test and no fix: a query whose semantics differ from what the caller assumes returns a plausible number rather than an error, and only a second method disagreeing will reveal it.

A citation that names more than one case

ambiguous-citation · found 13 September 2026 · fixed

The defect. Queries asking about a named case were built from the bulk citations table, which carries one row per cluster and never says how many clusters sit at a given reporter, volume and page. Where several cases share a citation, a query built on it asks about none of them in particular and has no single right answer.

Evidence. Two drawn queries failed at resolution, the resolver refusing to choose between the matches rather than picking one.

Citation	Cases matching
185 A.3d 1096	6
32 A.3d 836	3

These are not duplicate records of one decision, which the resolver already collapses. They are distinct cases sharing a page. Counting the whole pool afterwards showed the scale.

Metadata pool	Candidates	Share
Total	1,988,809	100%
Citation names more than one case	1,079,326	54.3%

What it caused. No published figure, because it was caught at resolution rather than after. It would have caused one. An under-specified query has no single correct answer, so a system answering it sensibly would have been recorded as wrong for a defect in the question rather than a fault of its own.

The uncomfortable part. Nothing was hidden. Every row needed to count clusters per citation was in the bulk table from the first download, and the pool build read that table without ever asking the question. This was not a limit of the data. It was a check nobody thought to write, and it stayed invisible for as long as nothing tried to resolve what had been drawn.

What fixed it. CitationRecord/tooling at b472838. The pool now records, for each candidate, whether its reporter, volume and page name more than one cluster, and the draw rejects on it, so the rejection appears in the record with its reason rather than being absorbed during the build.

Residual. The count describes one bulk generation. A later drop may split or merge clusters, so it has to be recomputed rather than carried forward. It also covers only the reporters the metadata pool draws from; the negated-holding category reaches its cases by a different path and is untouched by this.

Zeros are recorded without their cause

zeros-without-cause · found 9 September 2026 · open

The defect. A sampled volume returning no opinions may be a gap inside the range the corpus holds, or a volume past the end of what the corpus holds at all. The probe records the same zero for both. Every rate computed from those zeros mixes two quantities that mean opposite things, and nothing in the output lets a reader separate them.

Evidence. Two reporters from the same four-volume probe, generation 2026-06-30, in CitationRecord/tooling at 8e8e576 under results/.

Reporter	Draw zeros	Past the extent	Inside the extent
Cal. App. 5th	3 of 4	3	0
F. Supp. 3d	2 of 4	0	2

Cal. App. 5th returns three zeros because the frame drew volumes 50, 75 and 92 while the corpus stops at volume 39. Those zeros say nothing whatever about how well volumes 1 to 39 are held. F. Supp. 3d returns two zeros from inside its held range, where 218 of 700 declared volumes carry no citation. The first is an artifact of the draw. The second is the sparse district court coverage this probe was built to find, and it stands.

What it caused. The Cal. App. 5th rate, three of four, was repeated as a coverage figure in a published article and in correspondence with Free Law Project. It is not a coverage figure. The same reporter counted per cluster against its parallel reporter, and counted across its whole declared range, gives two further numbers. Three quantities, three populations, and the one that travelled furthest was the one least connected to coverage.

What would fix it. Not trimming the draw to the held range. That would hide precisely what the declared range exists to expose, and would turn a reporter the corpus barely holds into one that looks complete. The fix is to record the corpus extent per reporter, classify each zero as interior or beyond-extent, and report the two rates separately, so neither can be quoted as the other.

Not applied. It is a change to what the probe records, and the probe is to be re-run once after both open entries are fixed rather than twice.

What it does not affect. The per-cluster parallel-citation counts, which never use a volume sample. Those are exact over a complete generation and are recorded separately.

Declared volume ranges drift from the corpus, and nothing reports it

declared-range-drift · found 9 September 2026 · open

The defect. Each reporter’s volume range is a declared parameter rather than a fact read from CourtListener. That is deliberate and correct: asking the corpus where its volumes stop, and then sampling only that far, would make every reporter look complete. But nothing compares the declared range against what the corpus actually holds, so a parameter that is badly wrong is indistinguishable in the output from one that is well chosen.

Evidence. Same generation and commit as the entry above.

Reporter	Declared last	Last held	Drift
U.S.	603	606	corpus runs past the declaration
F.3d	1000	999	close
F. Supp. 3d	700	778	corpus runs past the declaration
Cal. App. 5th	100	39	declaration runs 61 volumes past the corpus

The drift goes both ways. Two reporters are held beyond their declared last volume, which means the frame samples a narrower span than the corpus offers. Cal. App. 5th overshoots by 61 volumes, which is what put three of its four draws past the end.

Why it is separate from the entry above. They have different fixes. The first is about what a zero means once it has been recorded. This one is about a parameter being wrong before any volume is drawn. A probe could classify its zeros correctly and still be sampling the wrong span, and correcting a range would not tell anyone what an existing zero meant. Folding them together would repeat the error both entries exist to record.

What would fix it. Report declared last volume against last volume held for every reporter, in the probe’s own output, and treat a large divergence as something to explain rather than something to absorb. The declared range stays declared. It just stops being unfalsifiable.

Not applied, for the same reason as the entry above.

What it does not affect. Anything that reads the citations table directly rather than through the frame.

A question about a judge who does not exist

authorship-undefined · found 18 September 2026 · fixed

The defect. One of the three metadata question types asks who wrote the majority opinion in a named case. It was asked of any drawn cluster. A trial court sits as a single judge, so it produces no panel, no majority, and no majority opinion. For such a case the question names nothing, and the recorded ground truth answered it anyway with the one judge’s name.

Evidence. An exploratory pilot put nine queries to four systems. One asked who wrote the majority opinion in a bankruptcy decision reported at 410 B.R. 170. Three of the four systems answered that a single-judge court produces no majority opinion. That is the correct answer. Our ground truth recorded an author, so a scorer applying it would have marked those three wrong and marked the fourth — which supplied a name — right.

The same item carried a second disagreement no scorer could have settled without going to the docket.

Field	Our ground truth	What the systems said
Court	District Court, W.D. Pa.	Bankruptcy Court, W.D. Pa.
Year	2008	2009
Judge	McVERRY	M. Bruce McCullough

What it caused. No published figure. It was caught because a pilot existed and because three systems disagreed with us in the same direction, which is the only signal that distinguishes a bad answer from a bad question. The registered query set for Edition One carried the same defect: the redraw rejected 1421350, a decision reported at 372 F. Supp. 671, for the same reason.

The uncomfortable part. This is the first entry here where the systems under test were right and the instrument was wrong, and where the instrument would have recorded its own error as their hallucination. Every other entry describes a number that could have been wrong. This one describes a question that had no correct answer, asked of four systems, three of which said so.

It is also the sharpest case for scoring against recorded ground truth rather than against a reading of the response. A scorer comparing a name to a name would have seen three failures. A scorer reading the responses would have seen three corrections.

What fixed it. Two checks in series, because neither is complete alone. At draw time the authorship question is refused for the reporters that are single-judge by construction — F. Supp., F. Supp. 2d, F. Supp. 3d, F.R.D. and B.R. — and for case names carrying a per curiam or order shape. That removes 196,192 of 1,988,809 metadata candidates, 9.9%, from the authorship question only; the year and citation questions are unaffected and keep the whole pool. At resolve time the question is refused again against the court the API actually reports and the per curiam flag, which catches the trial-court decisions that reach a regional reporter and which no field in the pool distinguishes.

Residual. The draw-time list is a list of reporters, not a parse of a court system, and the regional reporters carry trial-court decisions the reporter name cannot reveal. That is why the resolve-time check exists, and it is why an item refused there is left incomplete rather than answered: an incomplete item is visible in the artifact and blocks registration, while a wrong one is invisible and gets published.

A negation that governed half a sentence

clause-boundary · found 18 September 2026 · fixed

The defect. The negated-holding category inverts a judge-written parenthetical by deleting its negator. A guard was meant to reject any candidate that did not sit in a single clause, on the ground that one negator cannot govern two. The guard counted full stops and semicolons. Those are sentence boundaries. A comma and a coordinating conjunction open a second clause inside one sentence, and that shape passed.

Evidence. The pilot drew this parenthetical, which has one full stop and no semicolon:

holding that the “discovery rule” does not apply to civil penalty enforcement actions, and the statute of limitations starts running when the fraud occurs

Deleting the negator inverted the first clause and left the second untouched, producing a query asserting that the discovery rule does apply and that the clock starts when the fraud occurs. Those cannot both hold: the discovery rule is the doctrine that the clock starts on discovery. Three of the four systems reported the contradiction rather than answering; the fourth silently repaired the query and answered the repaired version, which is the one response that leaves no trace of the defect.

What it caused. No published figure. The three acceptable responses recorded for this category are that no such case exists, that a contrary case exists, or that a genuine superseding case exists. None of them covers “the question contradicts itself”, so the three systems that identified the defect precisely would have scored as failing to produce an acceptable response.

The uncomfortable part. The rule set for this category was already documented as empirical and incomplete, derived by reading draws rather than by reasoning about negation in advance, with a note predicting that further failure shapes remained in the pool. This is that prediction coming true for the sixth time. The note was right and it did not help, because a rule set can only be extended by the next failure finding someone.

What fixed it. A sixth rejection rule, on a comma followed by a coordinating conjunction, applied before the negator is counted. It removes 4,474 of 65,538 previously-accepted candidates, 6.8%, leaving 61,064. It is deliberately over-inclusive: it also rejects a serial list, where the single negator does govern the whole text and the inversion would have been clean. That is the trade this component always makes, for the reason it always gives — the pool is large enough that discarding is free, and a doubtful negation is not. The parenthetical above is pinned as a test, verbatim from the corpus, so a later loosening fails against the text that broke it rather than against a tidied version of it.

Residual. The rule set is still empirical and still not proven exhaustive, and this entry is evidence for that rather than against it. A seventh shape should be expected. The 2026.Q4 draw was re-run under the new rule and none of its five negated items was affected, so this defect reached the pilot and not the edition.

An answer recorded in a different form from the one the question asked for

answer-in-the-wrong-form · found 18 September 2026 · fixed

The defect. The citation question asks for a Bluebook-formatted citation for a named case. The recorded answer was the bare reporter citation the query had been built from — 513 B.R. 896. A Bluebook citation names the case, the reporter, the deciding court and the year. The two are not the same thing, and the question asked for the one the answer did not hold.

Evidence. The 2026.Q4 item asks for a Bluebook citation for Okke v. Okke (In re Okke). The complete citation is Okke v. Okke (In re Okke), 513 B.R. 896 (Bankr. W.D. Mich. 2014). The recorded answer held the middle third.

The pilot had already shown what this costs. Asked for a Bluebook citation for a Michigan case, three systems returned three strings, two of them carrying a parallel citation to the official reporter alongside the regional one. Compared to a single stored string, a correct response and an incorrect one are indistinguishable.

Source	Returned
Ground truth as recorded	403 N.W.2d 591
One system	People v. Lodge, 403 N.W.2d 591 (Mich. Ct. App. 1987)
Another	People v. Lodge, 157 Mich. App. 190, 403 N.W.2d 607
A third	People v. Lodge, 164 Mich. App. 558, 417 N.W.2d 582

What it caused. No published figure. It was found by reading the question and the answer against each other before registration, which is the only check that finds this class: the answer field was populated, well formed, and drawn from the corpus, so nothing that verified ground truth existed or resolved would have flagged it.

The uncomfortable part. This is the third entry in a row where the instrument asked one thing and recorded another, and the third where a correct response would have scored as wrong. The first two were caught by systems disagreeing with us. This one was caught by asking, of an item that had already passed every automated check, what question it was actually putting. That is not a check that can be automated, because the defect is agreement between a question and an answer, and both were individually valid.

What fixed it. The answer is now the complete Bluebook citation, assembled from resolved fields. The court is written in its Bluebook abbreviation rather than CourtListener’s court string, and the abbreviation is fetched rather than derived: Table T1 is long, and a mapping maintained in the tooling would be a second source of truth that drifts from the first. CourtListener’s courts endpoint carries citation_string, which is that abbreviation, and the lookup is recorded in the item’s provenance. The assembly refuses on any missing component rather than building a citation around a blank, because a partial citation looks like an answer.

The components are recorded beside the whole, so a scorer can check the case name, the reporter citation, the court and the year separately. A response carrying a parallel citation is not thereby wrong, and a ground truth holding one string cannot say which part of it differs.

Residual. Form normalisation is now the open question rather than form disagreement. A year recorded as 1981 and answered as “1981”, or an author recorded as Nieto and answered as Judge Nieto, are matching problems for the scoring interface rather than defects in the question. The other three metadata question types were checked against their answers at the same time and agree about what is being asked.

The public lookup journal discloses the items it verifies

lookup-journal-discloses-items · found 18 September 2026 · open

The defect. Ground truth for a drawn item is established by resolving its citation through resolve/, and every lookup is written to lookups.jsonl with the citation it looked up. That journal lives in CitationRecord/resolutions, which is public by design. Recording the ground truth of an item therefore publishes the citation the item is about, before the edition ships.

Evidence. Forty lines are already public, logged between 7 and 13 September 2026, carrying nineteen distinct citations. Eight of the nine items in the registered 2026.Q4 query set are among them.

Registered item	Subject citation	State
meta-author-2633759	176 P.3d 851	already public
meta-year-7579586	394 So. 2d 540	already public
meta-citation-8524823	513 B.R. 896	already public
neg-8018431	860 S.W.2d 414	already public
neg-6546281	568 N.W.2d 324	already public
neg-8816042	509 U.S. 43	already public
neg-7718500	409 U.S. 352	already public
neg-7023026	468 U.S. 420	already public
meta-author-5347747	39 N.Y.S. 959	withheld

The pending lines covering the ninth item are being held and will ship with the edition. Holding them protects one item of nine. The other eight were published days before the set they belong to was registered, and git history survives deletion, so they cannot be unpublished.

The design conflict underneath. This is not an oversight in one component. Two commitments this project has published are both correct, and for any item whose ground truth needs a lookup they cannot both hold.

Commitment	Says
resolve/	The lookup journal is public, because a verification nobody can read verifies nothing.
The homepage	The query set is pre-registered rather than published, and disclosed once the edition ships.

Verifying an item publicly and holding the item back are the same act performed on the same citation, and the journal cannot do one without undoing the other.

Why the guard did not catch it. register/ refuses to register a query set that sits inside a public repository, by name and by the generic git check, and that guard worked exactly as written every time it ran. It checks where the artifact goes. It has nothing to say about what a different file, in a different repository, reveals about the artifact's contents. The set was never in a public repository. Its subjects were.

What it caused. Nothing published, and nothing yet measured. What it removes is the held-out property of eight ninths of Edition One. A system whose training or retrieval has reached the public journal has seen which cases are being asked about, though not what is being asked about them.

This one does not tilt the way the others do. Every previous entry biases toward recording our own failure as somebody else's. This one biases the other way and is worse for it. Contamination makes a system look better, not worse: an item a system has already seen is one it answers correctly. The result is not a wrong number that could be corrected. It is a number that measures retrieval of this benchmark rather than retrieval of the law, and no scoring decision downstream can tell the two apart.

What would settle it, none of which is chosen here. The journal could record the resolution without the citation, deferring the query field for edition items until the edition ships, which keeps the record and withholds the subject. Edition lookups could go to a separate journal, private until the edition ships and published with it, at the cost of two journals and the rule about one home. Or the homepage sentence could narrow to what the design can actually deliver, which is that the questions and their answers are held out rather than the cases they are drawn from.

Residual, and it is the whole entry. The first option is the only one that restores the property for future editions, and none of the three recovers the eight items already disclosed. Those are spent for the held-out purpose whatever is decided next, in the same way the pilot's nine are spent: not by being answered, but by being seen.