The practical brief
The finding
Across 192 calculation traces on 96 selected financial questions, 27 matched the numerical answer key but cited unsupported financial roles. [c1] [c3] [c8]
Why it matters
A business can publish a correct calculation whose cited evidence does not substantiate the claim.
| Acceptance rule | Accepted traces | Joint passes |
|---|---|---|
| Signed-number matching plus execution/unit guards | 154 of 192 natural traces | 141 of 192 natural traces [c2] [c9] |
| Jev support gate plus execution/unit guards | 146 of 192 natural traces | 142 of 192 natural traces [c2] [c10] |
What to try
BLURSOR’s practical interpretation
Our interpretation: review the cited period, metric, entity and accounting status separately from arithmetic before publishing consequential financial claims.
Study boundary
This selected benchmark does not measure everyday error prevalence, AI discoverability or marketing performance.
The risk is a correct number with wrong support
Before publishing an AI-generated financial comparison, check whether its citations support the particular claim—not merely whether the calculation works. Financial reports repeat values across periods and accounting categories, so the right number can identify the wrong fact.
Researchers tested GPT-4.1-mini on 96 selected FinQA questions, a benchmark of calculations over financial reports. Two calculation traces per question produced 192 records. A trace records operands, arithmetic, source locations and an answer.
Source review identified 27 answer-key-matching traces across 14 questions with unsupported cited roles. Exact quotation checks caught all 27, but also rejected 33 supported traces. Strict text matching therefore had a cost: withholding valid evidence.
The AI verifier added little beyond numerical checks
The study evaluated Jev, an AI judge assigning a probability that cited evidence supports the requested financial roles. Its simpler comparator checked signed operand values at their cited locations. Both acceptance rules also required an available executed result and exact agreement between emitted and requested output-unit strings. Matching unit labels does not establish correct numerical scaling.
At an inclusive 0.8 support-probability cutoff, Jev accepted 146 of 192 traces, with 142 joint passes—supported citations plus correct executed answers. The numerical rule accepted 154, with 141 joint passes. Jev’s net gain of one joint pass depended on a single contextual-sign question.
That is a coverage tradeoff, not proof that an AI judge generally improves a business workflow.
Citation swaps exposed errors that numbers cannot reveal
Researchers then moved one citation to another cell containing the same number while keeping the question, operands, arithmetic and executed result fixed. Controls included alternative citations expressing genuinely equivalent facts.
In the initial constructed test, four of six clear wrong-role alternatives remained accepted at the 0.8 cutoff; two of six valid equivalent alternatives were withheld.
Repeating column headers explicitly corrected selected wrong-period and wrong-status decisions, but also reduced support for valid alternatives. A constructed follow-up on 36 new source pages, assessed by one non-author reviewer, showed the same tradeoff. Excluding a disputed accounting-status case removed its net paired-separation gain.
Separate publication checks from confidence scores
Our interpretation is to assess calculation accuracy and citation support separately. Explicit labels may help inspection, but these experiments do not establish a reliable optimization tactic.
The original numerical-release workflow did not improve accuracy and did not include the later source verifier or header intervention. The study cannot tell owners whether these changes improve AI discovery, rankings or live-agent performance.
- Check arithmetic, output units and numerical scaling independently.
- Inspect the cited metric, entity, period, accounting status and contextual sign.
- Evaluate both missed unsupported citations and rejected valid alternatives.
The source and evidence
Same-Number Citation Swaps: Stress-Testing Jev as a Financial Evidence Judge
Study limitations and disclosures
- The natural sample selected single-operation calculations. Historical labels came from unblinded assistant review; the follow-up used one non-author reviewer without inter-human reliability estimates or consensus adjudication. Source defects and disputed role interpretations affected sensitivity analyses.
- The verifier, numerical baseline and initial header experiment were exploratory. The follow-up used structurally filtered, program-derived records, not naturally occurring model errors. Technical repeats are not independent cases; results do not estimate real-world failure prevalence.
- Only one returned Jev identity and a GPT-4.1-mini serving alias were tested. Bootstrap intervals exclude annotation uncertainty and future serving variation. Inspection artifacts remain private, with public access pending.
- Authors list affiliations with Sofia University, Indiana University, the University of California, San Diego, Northeastern University and Michigan State University. The supplied paper provides no funding or commercial-interest declaration; this does not establish their absence. It acknowledges CodeX (OpenAI) assistance with editing, experiment running and script development, with authors responsible for verification and final text.
Evidence behind this briefing
[c1] A correct numerical answer did not guarantee a credible citation: 27 of 192 traces across 14 questions matched the answer key but cited unsupported financial roles. Exact quotation correspondence caught all 27, but also rejected 33 supported traces.
III-A Natural records: discrimination and a stronger numerical control · Read the study
[c2] At an inclusive 0.8 support-probability cutoff, Jev delivered only one more joint pass—supported citation plus correct executed answer—than signed-number matching, while accepting eight fewer records. The incremental joint-pass benefit depended on one contextual-sign question.
III-A, comparison following Table II · Read the study
[c3] The controlled citation-swap test isolated evidence-role recognition rather than arithmetic accuracy: it changed one citation to another cell containing the same number while keeping the question, source, operands, operation and executed result fixed.
III-B Same-number citations: isolating financial role · Read the study
[c4] In the initial constructed swaps, Jev missed four of six clear wrong-role contrasts at the 0.8 cutoff, including citations with support probabilities near 0.95. It also withheld two of six valid equivalent-fact alternatives. Strong discrimination on natural traces therefore did not establish reliable role recognition.
III-B, results following Figure 1 · Read the study
[c5] Explicit column headers changed verifier decisions in a controlled, selected-case experiment: all eight wrong-role swapped-citation calls changed from accepted to withheld at 0.8, while original citations remained accepted. This was a local representation effect, not proof of a general improvement; equivalent valid citations also lost support.
III-C Diagnosing selected failures through explicit column labels · Read the study
[c6] The 36-question follow-up used constructed records on new local source pages and one non-author reviewer. Among 15 pairs where only the original citation was supported, explicit columns eliminated two unsupported alternative acceptances but increased withholding of supported originals from five to six. The one-pair separation gain disappeared when a disputed issued/outstanding case was excluded.
III-D Fresh source pages with a returned non-author review, column-only subgroup and sensitivity analysis · Read the study
[c7] The prospectively specified numerical-release workflow did not improve accuracy: executed fallback reduced key matches by an average 1.5625 percentage points across 96 questions, with a descriptive 95% question-bootstrap interval spanning zero. This workflow did not incorporate the later source verifier or header intervention.
IV Verification and answer-release outcomes, following Table III · Read the study
[c8] Generalization and label reliability are limited: the natural bank selected single-operation calculations, historical labels were unblinded assistant judgments, and the follow-up used only one reviewer without inter-human reliability or consensus adjudication. Source defects and financial-role interpretation required judgment.
VI Limitations and reproducibility · Read the study
[c9] The relaxed-pointer numerical comparator required an available executed result and exact agreement between emitted and requested output-unit strings, in addition to signed-operand matching at the cited location. Unit-label agreement did not establish correct numerical scaling.
III-A, relaxed pointer rule definition preceding Table II · Read the study
[c10] The Jev source-probability acceptance counts in Table II likewise required available execution and matching output units in addition to the probability threshold; they were not probability-only acceptance counts.
III-A, paragraph introducing Table II · Read the study