The practical brief
The finding
A synthetic legal benchmark found that high overall verification accuracy could coexist with approval of unsupported claims. [c1] [c2] [c3] [c4] [c13]
Why it matters
Businesses reviewing AI-assisted legal content should distinguish an automated approval from evidence that a particular source supports a particular statement.
| Evaluated model | Overall classification accuracy | Not Found claims incorrectly approved |
|---|---|---|
| gemini-3-flash-preview | 97% across 3,396 benchmark claims | 3.67% of 849 Not Found benchmark claims [c1] [c3] [c4] |
| llama-3.3-70b | 81% across 3,396 benchmark claims | 48.76% of 849 Not Found benchmark claims [c1] [c4] |
What to try
BLURSOR’s practical interpretation
Our interpretation: check consequential claims against specific source passages and assess missing-support approvals separately from overall accuracy.
Study boundary
The experiment tested supplied court opinions and model-assisted synthetic claims, not business content, search visibility or deployed legal research services.
The decision: is automated approval enough?
Before publishing AI-assisted legal explanations or relying on them for a business decision, consider what an automated citation check actually establishes. This study identifies a risk beyond nonexistent sources: a checker can approve a plausible legal statement even when the cited opinion does not support it.
For businesses using citations to substantiate content, source existence and source support are different questions. The experiment tested the latter. It did not measure customer trust, brand discoverability or marketing performance.
What the researchers tested—and how they built it
The researchers built PARCEL, a benchmark containing 3,396 synthetic claims derived from 85 New York State Court of Appeals decisions issued in 2024–2025. Explanatory parentheticals—short explanations appended to legal citations—provided the writing style.
Seven models received a complete opinion and one claim, with no task-specific worked examples. They classified each claim as Supported, Refuted or Not Found: respectively, evidence supporting it, evidence contradicting it, or insufficient evidence either way. The dataset contained 849 Supported, 1,698 Refuted and 849 Not Found claims.
Construction was model-assisted. Gemini-3-flash-preview extracted examples from real briefs; an AI judge filtered examples; models generated supported and adversarial claims using the opinions. Gemini was also the highest-scoring verifier. That overlap warrants caution, but does not establish that Gemini generated every claim or that its construction role caused its ranking. Comprehensive independent expert validation of the final labels was not reported.
High accuracy did not eliminate unsupported approvals
Gemini-3-flash-preview achieved the highest reported overall accuracy, 97%, but incorrectly approved 3.47% of the 2,547 Refuted or Not Found claims. The authors call this false entailment: asserting source support where the benchmark label says none exists.
For llama-3.3-70b, overall accuracy was 81%, yet 48.76% of the 849 Not Found claims were accepted as Supported. An aggregate score therefore concealed a substantially different risk: endorsing claims whose evidence was absent.
Manual examination of 27 claims missed by at least six models found generally accurate legal statements accepted despite lacking support in the particular opinion. This qualitative analysis suggests that background knowledge can interfere with source checking; it is not a controlled causal test of model internals.
A realistic interpretation for content review
Our interpretation is to treat source support as a separate review question, rather than infer it from fluent wording, a genuine citation or a strong model score. Automated verification can assist review without becoming the final authority for consequential legal content.
- Ask reviewers to identify the passage supporting each consequential claim, including its conditions and scope.
- Separate “contradicted” from “not established by this source”; missing support does not necessarily make a statement legally false.
- When evaluating a checker, examine unsupported-claim approvals alongside overall accuracy. These suggestions were not tested as interventions.
What the study cannot tell your business
These are not estimates of error rates in ordinary business answers. The synthetic claims had a fixed class balance, and the corpus excluded short opinions and opinions triggering safety filters. Providing the full opinion isolated verification, rather than testing retrieval or answer generation.
The strict grounding rule can label an accurate statutory paraphrase Not Found when its descriptive language is absent from the opinion. Model rankings should therefore be read as performance against this benchmark, not definitive expert-certified legal judgments.
The study establishes no improvement in visibility, recommendations or commercial performance. Multiple-model checking was proposed but not experimentally validated. The authors disclose a public dataset and prompt repository; availability alone does not validate its labels.
The source and evidence
When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection
Study limitations and disclosures
- The benchmark uses model-generated synthetic claims with a deliberately fixed 1:2:1 Supported:Refuted:Not Found balance, not a representative sample of naturally occurring AI errors.
- Coverage is limited to 85 New York State Court of Appeals decisions from 2024–2025. Opinions shorter than 2,000 characters and those triggering model safety filters were excluded.
- Models received the full opinion. Results do not measure retrieval quality, end-to-end answer generation, commercial content accuracy or business discoverability.
- Construction involved Gemini extraction, AI-judge filtering and synthetic claim generation. Comprehensive independent expert validation of the final labels was not reported. Gemini-3-flash-preview also led the verifier ranking, but the study does not establish that construction overlap caused its performance.
- Strict source grounding differs from general legal truth: an accurate statutory paraphrase may be labeled Not Found when its descriptive language is absent from the opinion. The qualitative error analysis does not establish a causal explanation of model behavior.
- Table 6's reported strategy averages are inconsistent with the averages calculated from its seven displayed model rows. This briefing therefore does not use those disputed averages in its comparison visual.
- Overall performance metrics are rounded to two decimals, with no reported confidence intervals.
- M. Mikail Demir and M. Abdullah Canbaz are affiliated with the University at Albany, SUNY. Research received partial support from SUNY System Administration through the SUNY AI Platform. The supplied paper contains no commercial-interest declaration; that absence does not establish that no commercial interests exist.
- The authors report a public repository containing the dataset and generation and evaluation prompts. Its contents were not independently inspected for this briefing.
Evidence behind this briefing
[c1] The benchmark tests source-grounded classification, not answer generation or retrieval quality: each model receives the full judicial decision and one claim in a zero-shot, three-label verification task. The 3,396 claims comprise 849 Supported, 1,698 Refuted and 849 Not Found instances.
Section 3.4, Verification Setup; sample counts in Section 3.3.3 and Figure 2 · Read the study
[c2] The sample is restricted to recent New York appellate opinions and excludes short decisions and decisions triggering safety filters. Those exclusions limit coverage of legal subject matter and do not establish performance on commercial content or general AI answers.
Section 3.3.2, Creating Synthetic Supported Claims from Judicial Decisions · Read the study
[c3] Among seven tested models, gemini-3-flash-preview had the highest reported overall classification accuracy, 97%, and macro F1, 0.97; kimi-k2-thinking reached 95% accuracy and glm-4.7 reached 94%. These are benchmark accuracy results, not preference or causal marketing effects; Table 2 rounds metrics to two decimals and supplies no confidence intervals.
Section 4.1, RQ1: Reliability of Hallucination Detection; Table 2 · Read the study
[c4] High accuracy did not eliminate false claims of source support. For llama-3.3-70b, 48.76% of the 849 Not Found claims were incorrectly accepted as Supported despite 81% overall accuracy. gemini-3-flash-preview had the lowest overall false entailment rate, 3.47%, measured among the 2,547 Refuted or Not Found claims.
Section 4.1; Table 3; denominator counts in Section 3.3.3 and Figure 2 · Read the study
[c5] Phantom Citation was the hardest strategy on average: accuracy averaged 0.72 across seven models on 330 claims introducing relevant but uncited precedents, versus 0.81 for 247 Missing Detail claims and 0.80 for 272 Unaddressed Argument claims. This measures detection of unsupported attribution, not the effect of citations on visibility or trust.
Table 6, Accuracy metric for not found-claim strategies; column order Phantom Cit., Missing Det., Unaddr. Arg.; strategy definitions in Section 3.3.3 and counts in Figure 2(a) · Read the study
[c6] Manual inspection of the 27 claims missed by at least six of seven models found cases where models accepted generally accurate legal statements that the particular opinion did not support. This is qualitative error analysis suggesting interference from prior knowledge, not a controlled causal test of model internals.
Section 4.3, RQ3: Systematic Failures and Error Diagnosis · Read the study
[c7] The authors disclose a public repository containing the dataset and templates for supported, refuted and not-found claim generation and model evaluation. Repository availability is reported by the paper; the supplied text does not itself expose those templates for inspection.
Appendix A, Prompts and Dataset; dataset availability also disclosed in footnote 2 · Read the study
[c8] Material funding disclosure: the research received partial support from SUNY System Administration through the SUNY AI Platform. This is a support disclosure, not evidence of sponsor influence on the findings.
Section 7, Acknowledgments · Read the study
[c9] The authors used google/gemini-3-flash-preview to extract parenthetical explanations and cited cases from 85 appellate briefs, yielding over 700 candidates. This is the same named model as the highest-scoring evaluated verifier reported in c3; the paper does not establish that this construction overlap caused its performance.
Section 3.3.1, Harvesting Parentheticals from Real Briefs; verifier ranking in Section 4.1 and Table 2 · Read the study
[c10] Following manual inspection of harvested parentheticals, an LLM judge filtered exemplars by reasoning density. The resulting 180 claims served as few-shot examples for synthetic generation; this reported inspection and filtering is not comprehensive independent expert validation of benchmark labels.
Section 3.3.1, Harvesting Parentheticals from Real Briefs · Read the study
[c11] The 849 Supported claims were generated synthetically using model prompting informed by harvested examples, with up to ten claims requested for each of 85 decisions. The passage does not identify the generation model, so it does not establish that Gemini generated every benchmark claim.
Section 3.3.2, Creating Synthetic Supported Claims from Judicial Decisions · Read the study
[c12] Refuted and Not Found adversarial claims were also generated with an LLM conditioned on the full judicial opinion, rather than by mutating claims without the source.
Section 3.3.3, Creating Refuted and Not Found Claims · Read the study
[c13] The paper describes dataset validation as a baseline model-reliability benchmark, but does not report comprehensive independent expert validation of the resulting labels. Its reported manual work covers harvested-example inspection and examination of 27 difficult instances, not an independently validated full-label audit. Model rankings should therefore be interpreted as performance against this model-assisted synthetic benchmark, not definitive expert-certified support judgments.
Section 3.4, Verification Setup; reported manual inspection in Section 3.3.1 and 27-instance error analysis in Section 4.3; Appendix A · Read the study