The practical brief
The finding
In a German statute retrieval test, fine-grained structured passages outperformed fixed 100-word chunks on citation-retrieval efficiency, but coarse structured passages did not. [c1] [c3] [c4] [c5] [c7]
Why it matters
Businesses evaluating a document-grounded AI system should distinguish finding relevant sources efficiently from producing correct, supported answers.
| Chunking scheme | Code-depth retrieval | Section-depth retrieval |
|---|---|---|
| Fixed 100-word chunks | 4.19 ToER per 1,000 mean-chunk words; top-five, code matching | 1.10 ToER per 1,000 mean-chunk words; top-five, section matching [c2] [c3] |
| Numbered-item chunks | 14.23 ToER per 1,000 mean-chunk words; top-five, code matching | 3.25 ToER per 1,000 mean-chunk words; top-five, section matching [c2] [c3] |
| Whole-section chunks | 3.06 ToER per 1,000 mean-chunk words; top-five, code matching | 0.85 ToER per 1,000 mean-chunk words; top-five, section matching [c2] [c4] |
What to try
BLURSOR’s practical interpretation
Our interpretation: ask vendors to test passage size and count appended context in efficiency comparisons, then evaluate generated answers separately.
Study boundary
The study tested retrieval, not business visibility, generated-answer accuracy or hallucination reduction.
Do not buy retrieval gains as answer-quality gains
If you are choosing an AI assistant that searches legal or policy documents, a claim of better citation retrieval is not enough to establish better advice. This study supports a narrower purchasing question: does the system find useful source passages efficiently, or simply supply more text?
For owners and marketers, the distinction also limits what this research says about discoverability. It does not show that restructuring a website will increase brand mentions in public AI answers. Its practical relevance is to evaluating document-grounded systems, especially where users need to inspect the sources behind a response.
What the researchers actually tested
The researchers evaluated retrieval against judicial citations in 12,037 held-out German Federal Court of Justice cases. Case-fact narratives served as queries. A single embedding model—a system that represents text numerically for similarity search—retrieved statutory passages.
They compared six ways of splitting documents into chunks, meaning separately searchable passages: fixed 100-word blocks and five statutory subdivision levels. A knowledge graph, a map of document units and their relationships, supported structured splitting and optional one-hop expansion: adding a retrieved unit’s parent text, cited text or both.
The main comparisons retrieved the top five chunks. The statute graph covered 90.3% of ground-truth citations; scoring excluded citations whose codes were absent from it. This was therefore a test within a bounded source collection, not comprehensive legal research.
Smaller units helped; extra context carried a penalty
Numbered-item chunks performed best when matching codes and statutory sections; sentence chunks led at deeper matching levels. Selected comparisons were roughly three times the fixed 100-word baseline. However, whole-section chunks underperformed that baseline at both code and section matching depths. Structure alone was not sufficient.
The score needs careful reading. Token efficiency recall, or ToER, scales citation-prefix recall per 1,000 words of mean corpus-wide chunk length. Despite its name, its denominator uses words, not tokens. It is neither an answer-accuracy percentage nor a direct measurement of total context for each query.
Adding related text initially appeared helpful when its length was not charged. Once appended context entered the denominator, every tested expansion strategy lost efficiency against its own unexpanded baseline at subsection matching depth. For subsection chunks with parent expansion, average delivered context increased from 70.6 to 415.0 words and ToER fell from 1.36 to 0.36.
A realistic interpretation for a business buyer
Our interpretation is to treat passage granularity as a testable system-design choice, not a universal rule to make every passage shorter. Preserving meaningful document boundaries looks worth comparing with fixed-length splitting in your own collection; this paper does not prove the winning configuration will transfer.
Ask for separate evidence on retrieval and final answers. A system can recover a relevant legal section without establishing that its response applies that section correctly.
- Compare chunking options on representative questions rather than accepting “graph-based” as a quality guarantee.
- Ask whether efficiency calculations include all appended context.
- Have qualified reviewers assess whether the retrieved material actually supports the generated response.
What the study cannot tell you
Results became substantially weaker as citation matching became more precise. The authors attribute this to the gap between factual narratives and abstract statutory language, but that explanation was not separately tested.
The study used one embedding model and did not validate downstream answer quality, hallucination reduction or legal-specialist judgments of sufficiency. Prefix matching also credits unspecified deeper subdivisions as implicit hits, so these scores should not be read as exact citation correctness at every level.
The source and evidence
Verifiable Retrieval for Legal Reasoning through Structure-driven Knowledge Graph
Study limitations and disclosures
- This is a German statute retrieval evaluation using one embedding model, not a test of brand visibility, marketing content or public AI-answer discoverability. Generated-answer quality and hallucination reduction remain unvalidated.
- The graph covers 90.3% of evaluation ground-truth citations. Scoring excludes citations whose codes are absent from the graph, so results do not represent retrieval of all cited law.
- Citation matching credits unspecified deeper subdivisions as implicit hits. The primary efficiency denominator is mean corpus-wide chunk length, rather than actual total context per query; expansion results change when appended context is counted.
- More precise citation matching remains difficult. The authors’ explanation involving the gap between case narratives and statutory language is an interpretation, not a separately established causal finding.
- The source lists Technische Hochschule Ingolstadt, Germany, as the authors’ institutional address. The supplied paper contains no funding statement or commercial-interest declaration; that absence should not be treated as confirmation that no funding or interests exist.
- The authors disclose using Claude, Gemini and Quillbot AI for language editing and generating LaTeX code for Figure 1 using non-generated data. They state that they reviewed and edited the content and take responsibility for it.
Evidence behind this briefing
[c1] The evaluation compares chunk granularity and one-hop retrieval on 12,037 held-out BGH cases. Its statute graph covers 90.3% of ground-truth citations; the remaining 9.7% are out of scope or unretrievable, and the metric excludes citations whose codes are absent from the graph.
Section 1.1, contribution 1; evaluation sample in contribution 3; denominator restriction in Section 3.2.3 · Read the study
[c2] Citation matching uses hierarchical prefix agreement, counting unspecified deeper levels as implicit hits. ToER measures prefix recall scaled per 1,000 words of mean corpus-wide chunk length, rather than actual total context per query; results are micro-averaged across cases. The expansion-cost analysis separately changes that denominator to include appended context.
Section 3.2.3, Citation Matching and Evaluation Metric; Equation (1); Section 4, Figure 3(b) discussion · Read the study
[c3] At top-5 retrieval, Nummer chunks perform best at Code and § matching depths, while Satz chunks perform best at deeper depths. Selected ToER comparisons are roughly three times the naive 100-word baseline. These are retrieval-efficiency scores per 1,000 words, not percentages of accurate AI answers.
Section 4, first paragraph; Table 1, top k=5 · Read the study
[c4] Preserving structure alone does not guarantee higher retrieval efficiency: coarse § chunks underperform the naive 100-word baseline at both Code and § matching depths. Granularity matters within this particular legal retrieval system.
Section 4, second paragraph; Table 1 · Read the study
[c5] At Absatz matching depth and top-5 retrieval, all tested one-hop expansion strategies lose efficiency relative to their unexpanded baselines when appended context is charged. For Absatz chunks with parent expansion, ToER falls from 1.36 to 0.36 as average delivered context grows from 70.6 to 415.0 words.
Section 4, Figure 3(b) discussion and Figure 3 caption · Read the study
[c6] Even the strongest configurations lose substantial efficiency as matching becomes more precise. The authors attribute this difficulty to the gap between factual case narratives and abstract statutory language; this attribution is not a separately tested causal result.
Section 5, Limitations · Read the study
[c7] The study does not validate whether granular chunking improves downstream generated answers or reduces hallucinations. Multiple-embedder comparisons, varied top-k tests, and legal-specialist judgments of retrieval sufficiency are proposed future work, not completed validations.
Section 5, Future Work · Read the study
[c8] The authors disclose using Claude, Gemini, and Quillbot AI for language editing and LaTeX-based Figure 1 production from non-generated data. They state that they reviewed and edited the content and take responsibility for it.
Declaration on Generative AI · Read the study