The practical brief
The finding
Across 20 tested models, strong answer scores did not reliably translate into answers supported by the cited PDF region. [c1] [c3] [c4] [c13] [c14]
Why it matters
Businesses using AI to summarize reports or substantiate claims need to check the evidence, not just the conclusion.
| Tested model | Answer correctness | Strict Attributed Accuracy |
|---|---|---|
| GPT-5.4 | 87.1/100 answer score in the PDF benchmark | 59.0/100 SAA score in the PDF benchmark [c3] [c4] |
| Gemini-3-Flash | 84.5/100 answer score in the PDF benchmark | 65.4/100 SAA score in the PDF benchmark [c3] [c4] |
What to try
BLURSOR’s practical interpretation
Our interpretation: assess answer quality and citation support separately, using document and question types representative of your work.
Study boundary
This supplied-PDF benchmark does not measure public AI search visibility, customer trust or brand recommendations.
A citation should not end your review
If you are choosing an AI tool to summarize business documents or support a published claim, a correct-looking answer with a citation is not enough. The cited passage, table or image may not support the answer—even when the answer itself is right.
That is the risk examined by CiteVQA. It tests evidence attribution: whether a model connects its answer to the specific document region that supports it. It does not test which businesses appear in public AI answers.
The test checks answers and supporting regions
The benchmark contains 1,897 questions across 711 English and Chinese PDFs in seven domains. Documents average 40.6 pages. Tasks involve one or multiple documents, and nearly 30% of designated crucial evidence elements are tables, images or equations rather than text.
The researchers evaluated 20 models, requiring answers with bounding-box citations—coordinates identifying a region on a page. Strict Attributed Accuracy, or SAA, judges answer quality and evidence support together.
SAA credits a sample when answer correctness reaches at least 4 out of 5, alongside either citation relevance of at least 4 out of 5 or crucial-evidence recall of at least 0.6. Recall measures how much designated evidence was recovered. The relevance route can credit genuinely supporting citations outside the designated regions; matching every reference box is not required.
Wrong evidence matters more than imperfect boxes
GPT-5.4 recorded an answer score of 87.1 but an SAA score of 59.0. Gemini-3-Flash recorded 84.5 and 65.4, respectively. These are different metrics on a 100-point reporting scale, not interchangeable accuracy percentages.
The highest overall SAA was 76.0 for Gemini-3.1-Pro-Preview. Rankings need caution: proprietary and open-source systems received inputs with unequal visual fidelity.
Among SAA-failing samples, wrong-page and wrong-element citations together accounted for 34.8%–51.7% across models. Boundary imprecision accounted for only 1.0%–5.0% of failures. Often, the problem was evidence selection rather than slightly inaccurate box drawing.
Performance also varied by document domain and question type. Appendix results identify Multimodal Parsing—locating an element from descriptive cues and reading its content—as a particular bottleneck. These are benchmark subgroup findings, not proof that a particular layout causes failure.
Make traceability a separate buying criterion
Our interpretation is to make evidence support a separate acceptance check for document-answering tools. A polished summary or high answer score alone is insufficient when another person must verify the claim.
- Inspect whether consequential answers point to supporting pages and regions.
- Pilot representative document domains and question types, including tasks that locate and read tables or images.
- Include multi-document questions if your workflow compares reports: attribution performance was lower in the benchmark’s multiple-gold-document setting.
- Keep input conditions comparable across vendors; image fidelity can affect evidence localization.
What this study cannot tell your business
These findings do not establish a tactic for earning more AI citations, improving customer trust or increasing sales. Nor do they predict any model’s performance in your deployed workflow.
AI models helped construct and score the benchmark, with human checks on sampled outputs rather than every item. Public PDFs may have appeared in training. Treat the results as a reason to test traceability on your own documents, not as a universal vendor ranking.
The source and evidence
CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence
Study limitations and disclosures
- The benchmark tests evidence localization within supplied PDFs, not public AI search discoverability or commercial outcomes. Performance varied across document domains and question types; Multimodal Parsing was a particular bottleneck. These subgroup findings do not establish causal layout effects or deployed performance.
- Generative AI assisted dataset construction, scoring, manuscript editing and code implementation. Human validation covered sampled outputs. A 200-sample judge comparison found no statistically significant scoring differences, but that does not prove equivalence or universal judge accuracy.
- Questions without an individually crucial evidence element were excluded, limiting the evidence structures represented.
- Unequal input resolutions and interfaces complicate cross-model comparisons. Public source PDFs may have appeared in training; a private post-cutoff evaluation was planned, not reported as completed.
- The authors acknowledge substantial curation and evaluation compute costs and unresolved domain-specific definitions of authoritative evidence.
Evidence behind this briefing
[c1] The benchmark contains 1,897 questions over 711 PDFs in seven domains, averaging 40.6 pages per document. Questions include single-document and two multi-document scenarios; Table 1 reports 451 English and 260 Chinese documents.
Section 3.4, Dataset Overview and Analysis; Table 1 · Read the study
[c2] Crucial evidence is defined through model-based, one-element-at-a-time masking. Questions with no individually crucial element are excluded, so the benchmark does not cover all possible evidence structures.
Section 3.3, Relevance Filtering and Crucial Evidence Identification · Read the study
[c3] Strict Attributed Accuracy is a sample-level binary threshold metric: answer correctness must be at least 4/5 and either citation relevance at least 4/5 or crucial-evidence recall at least 0.6. Thus, SAA can credit supporting citations outside the designated crucial evidence; it does not require every ground-truth box to be matched.
Section 4.1, Evaluation Metrics · Read the study
[c4] In the 20-model PDF screenshot evaluation, GPT-5.4 scored 87.1 for answer correctness versus 59.0 SAA, and Gemini-3-Flash scored 84.5 versus 65.4. These are different metrics on a 100-point reporting scale, not two interchangeable accuracy percentages. Table 2 ranks Gemini-3.1-Pro-Preview highest overall at 76.0 SAA and Qwen3-VL-235B-A22B highest among tested open-source models at 22.5.
Section 4.3, The "Attribution Hallucination" Phenomenon; Table 2; Section 4.2 · Read the study
[c5] Among SAA-failing samples, boundary imprecision accounts for 1.0%–5.0% across models, while wrong-page and wrong-element citations together account for 34.8%–51.7%. The denominator is failures, not all benchmark questions; the authors distinguish evidence-selection failures from box-drawing failures.
Section 5.1, What Drives SAA Failures?; Figure 5 · Read the study
[c6] The authors disclose generative-AI dataset construction and evaluation: models extract evidence, synthesize and paraphrase questions, verify answerability, and identify crucial evidence; Qwen3-VL-235B-A22B judges correctness and relevance. Human validation covers randomly sampled outputs rather than every item. AI also assisted manuscript editing and code implementation.
AI use statement · Read the study
[c7] In a Qwen3-VL-235B-A22B input-resolution experiment, halving the pixel budget reduced SAA from 22.5% to 11.8%. This is evidence-attribution performance under a specific visual-input intervention, not a measured effect on brand visibility or trust.
Appendix C.1, Table 10 and Trade-off Analysis of Input Resolution · Read the study
[c8] Cross-model performance comparisons are confounded by unequal input fidelity: proprietary systems receive native or full-resolution pages, while open-source models receive downsampled inputs constrained by context windows.
Appendix C.1, Trade-off Analysis of Input Resolution · Read the study
[c9] In 200 randomly selected samples, automated relevance and answer-quality scores did not differ significantly from human expert scores in the reported Friedman tests. Mean differences were 0.02–0.16 on a five-point scale; nonsignificance is not proof of equivalence or a general guarantee of judge accuracy.
Appendix C.2, Validation of Automated Evaluation via Human Study; Table 11 · Read the study
[c10] Attribution performance was lower in Multi-Doc (N-Gold) than Single-Doc settings: GPT-5.4 page recall fell from 88.5% to 75.4% and bounding-box F1 from 29.6% to 20.6%; Qwen3-VL-235B-A22B page recall fell from 64.4% to 50.5%. These are benchmark scenario comparisons, not demonstrated causal effects in deployed search systems.
Appendix C.4, Impact of Multi-document Complexity; Table 12 · Read the study
[c11] Auxiliary training validation compared 3,000 CiteVQA-generated samples with 5,000 originally human-annotated samples from the same ViDoRe V3 PDFs, using AgenticOCR-style trajectory distillation and Qwen3-VL-8B-Instruct fine-tuning. Evaluation used approximately 200 FinRAGBench-V bounding-box samples and approximately 400 disjoint, manually annotated ViDoRe V3 test samples. Results were mixed: broadly comparable on FinRAGBench-V, with an advantage for original data on ViDoRe V3; this does not establish universal superiority of synthetic training data.
Appendix B.4, Details of Auxiliary Training Validation; Table 8; Trajectory Distillation via Rejection Sampling; Training Setup; Model Evaluation · Read the study
[c12] The authors disclose unresolved domain-specific definitions of authoritative evidence, substantial curation and evaluation compute costs, possible training exposure to public source PDFs, and unequal visual-input interfaces. A private post-cutoff evaluation subset is planned, not reported as completed.
Appendix F, Limitations · Read the study
[c13] CiteVQA benchmark attribution performance varied across document domains. Appendix C.5 reports peak performance in Academic Tech, including an SAA score of 85.0 for Gemini-3.1-Pro-Preview, versus a highest SAA of 63.3 in Publishing & Media. These are benchmark subgroup results, not demonstrated causal effects of layout or predictions of deployed performance.
Appendix C.5, More Fine-grained Results — Document Type; Figure 8 (SAA Score) · Read the study
[c14] CiteVQA benchmark performance also varied by question type: Quantitative Reasoning was a strength, with Gemini-3.1-Pro-Preview scoring 82.6 SAA, while Multimodal Parsing was identified as a major bottleneck involving difficulties in both precise evidence attribution and answer generation.
Appendix C.5, More Fine-grained Results — Question Type; Figure 9 (SAA Score) · Read the study