Check the Source Review Behind an AI Quality Score

An answer can be fluent, relevant and easy to read while citing a source that a subject expert would reject. That makes source review a separate job from judging whether the answer sounds useful.

A prelaunch evaluation of Evrópuvefur, a University of Iceland public-information service, offers a concrete example. The service answered the same set of 287 questions using either a curated collection or open-web search. Experts then evaluated a subset of the resulting answers and their sources.

Among 187 reviewed web-search answers, 65 contained at least one source flagged for untrustworthiness or irrelevance: about 35%. That is a meaningful finding about the reviewed web answers. It is not a general error rate for web-enabled AI.

The paper also reports flags in 16 of 262 curated-retrieval answers, about 6%. Those curated sources were checked for currency, while web sources were checked for trustworthiness and relevance. The two rates therefore cannot establish that curated retrieval was roughly six times safer or more trustworthy.

Read the label on the failure count

A percentage is only comparable when the underlying event is comparable. “Contains an outdated source” and “contains an untrustworthy or irrelevant source” are different events.

Imagine testing one set of pages for broken links and another for incorrect prices. Even with the same number of pages, a lower failure rate in the first test would not prove better overall page quality. The tests looked for different problems.

The same issue applies here. The difference in flagged failure types was partly built into the evaluation instructions. It should not be described as though experts applied identical checks and discovered that each retrieval mode had a different characteristic weakness.

A fair trust comparison would require the same relevant criteria on both sets, with comparable review procedures. The existing study still provides useful information: experts identified source concerns in a substantial share of the reviewed web-search answers. That finding stands without turning the curated comparison into a safety ranking.

Answer quality and source quality need separate records

The study used five domain experts, a seven-criterion answer rubric and source-level flags. Across the evaluated material, it collected 551 evaluations of 449 generated answers.

This separation is valuable. An overall answer score can conceal a specific source problem. A response might answer the question coherently while using an unreliable publication, or cite a trustworthy institution on a page that does not support the claim.

For your own evaluation, record the claim, cited page and reason for concern together. “Bad citation” is too broad to repair. An outdated policy needs a current source; a source unrelated to the claim needs a different evidence connection; an unreliable publisher raises a different question again.

The expert flags in this study are descriptive judgments, not a final adjudication of every source. Human review also needs clear criteria and a way to resolve disagreements. Calling a process expert review does not make its categories interchangeable.

A preferred-domain prompt is not an allowlist

The researchers also tried steering web search toward trusted domains. Citations to listed domains increased from 12% to 21% in the reported prompt comparison.

That shows a change in the citation mix. It does not show that the remaining sources were unacceptable, that every listed-domain citation supported its claim, or that the instruction enforced a hard retrieval boundary.

If your service must use only approved sources, inspect how that restriction is implemented. A request in a prompt and a retrieval system technically limited to an approved collection are different controls. Even a genuinely restricted collection still needs maintenance and checks that the selected passage answers the question.

Define the review before choosing the winner

For a customer-facing assistant, start with the questions it must handle and the source failures that would matter. Apply the same relevant source criteria to the retrieval approaches you compare. Keep answer coverage, factual support, source relevance and currency visible as separate outcomes.

When one approach cannot answer a question, record that too. A system that declines outside its collection and one that answers from the web make different trade-offs; an average score can hide them.

The first check is straightforward: open a sample of answers that scored well and read their cited pages. Confirm which claim each page supports. That tells you whether the quality score includes the evidence judgment you thought it did.