AI Systems Often Disagree on Facts, Study Finds
A new study reveals that when asked to assess the truth of a statement, five of the world’s most advanced AI systems frequently provide conflicting answers. The research, published this month by Kosta Jordanov at Lenz Research, found that at least one AI model offered a different verdict in two-thirds of the cases examined.
The study evaluated GPT-5.4, Claude Opus 4.7, Gemini 3 Pro, Gemini 3 Pro with Search, and Sonar Pro on 1,000 real-world fact-check claims submitted by actual users. The models were tasked with selecting one of four labels: true, mostly true, misleading, or false.
In 672 out of the 1,000 claims, at least one model diverged from the majority opinion. The disagreement was significant in 34% of the cases, with one model labeling a claim as true while another deemed it false.
The study emphasized that the claims were submitted by real users for verification, rather than drawn from standard test sets. “Only one verdict bucket can be correct per claim, so any disagreement among the panel means at least one model’s verdict is label-inconsistent under this 4-bucket rubric,” the study stated.
This research highlights a different challenge than the previously documented issue of AI “hallucination,” where chatbots fabricate information. Instead, the study reveals that these models struggle to agree on basic factual judgments about the same material.
The researchers employed a methodology designed to make it more difficult for AI companies to dismiss the findings. The claims were sourced from real-world submissions to Lenz’s fact-checking platform, making it unlikely that they would appear in training data with a definitive answer key. The study noted, “Most of these claims are unlikely to appear in any training corpus with a gold label attached—there’s no canonical answer key to pattern-match against, no benchmark leaderboard to anchor to.”
The statistical measure of agreement, Krippendorff’s alpha, was 0.639 on a scale where 1.0 represents perfect agreement and 0 represents random chance. The study indicates “nontrivial but limited agreement.” Researchers noted, “The models’ verdicts are structured rather than random, but not consistent enough to treat the panel as a single interchangeable judge.” Generally, researchers consider anything below 0.8 to be weak.
When all five models did agree, which occurred in only 328 out of 1,000 claims, they rarely agreed that a statement was misleading or mostly true. Only four claims received a unanimous “misleading” verdict, and none received a unanimous “mostly true” verdict.
Examples of Disagreement
The researchers provided examples of claims where the AI models showed the most divergence. For instance, regarding the claim “The World Bank’s active portfolio in Nigeria stands an over $16.4 billion as of 2025,” GPT-5.4 labeled it “mostly true,” while Gemini 3 Pro called it “false,” and Gemini 3 Pro + Search rated it “misleading.”
In another example, the models were presented with the claim: “Donald Trump said that an attack on Iran was postponed at the request of Gulf Allies.” GPT-5.4 said it was false, Claude Opus 4.7 called it mostly true, Gemini 3 Pro said false, and Gemini 3 Pro + Search rated it true.
The researchers found that the models tended to converge on definitive verdicts, with disagreement occurring in the middle ground of the rubric. Unanimity was only observed at the extremes: either the claim was definitively true or definitively false.
Implications for Fact-Checking
This finding is particularly relevant as people increasingly rely on AI systems for fact-checking. The study suggests that users could receive conflicting answers from different AI models when verifying a claim, raising questions about which source to trust.
The study also challenges the narrative of continuous improvement often promoted by AI companies, which frequently cite benchmark scores. The Lenz study tested the models on the kind of ambiguous claims that humans commonly debate, revealing significant disagreement among the AI systems.
The paper cautions against assuming that the majority verdict is always correct. “A majority of frontier models is not ground truth. The majority verdict is sometimes wrong; an individual dissenting model is sometimes right. We use the majority as a structural reference point for measuring disagreement, not as a stand-in for correctness.”
The study highlights a deeper issue: when models disagree, at least one must be incorrect. The study calls a model’s verdict “label-inconsistent under this 4-bucket rubric.” There is no mechanism for resolving these disagreements.
On the 328 claims where all five models agreed, zero received a unanimous “mostly true.” The nuance bucket emptied out completely. If AI models can only find consensus at the extremes, can they be trusted as fact checkers at all?
Leave a Reply