NPR and NewsGuard tested six major AI chatbots using 15 false narratives linked to state-aligned actors from Russia, China and Iran. The systems correctly rejected the misleading claims about 75% of the time, outperforming conventional first-page search results. But the findings also show that web access does not guarantee reliability, with source quality remaining a decisive weakness.
Can chatbots be more resistant to propaganda than the search engines feeding them information? A joint NPR and NewsGuard experiment produced that counterintuitive result. Given 30 questions built around false narratives pushed by actors linked to Russia, China and Iran, six web-enabled AI assistants correctly debunked the narratives about 75% of the time.
The mid-July test covered ChatGPT, Gemini, Copilot, Meta AI, Grok and Claude. NewsGuard selected 15 false narratives that had emerged between December 2025 and July 2026. Then researchers created two questions for each: one neutrally asking whether an event had occurred, and another written as though the false premise were already true. Responses were scored against NewsGuard fact checks. NPR measured whether a system clearly challenged a false premise and ultimately reached the correct conclusion. It did not independently verify every sentence the models generated.
Within those limits, chatbots failed less often than first-page conventional search results. One reason appears to be that conversational systems can do more than retrieve links: they can compare sources, identify their provenance and explicitly tell a user that the assumption embedded in a question is false.
The weaker layer was AI embedded in search
The picture changed when NPR examined the automated summaries increasingly placed above search results. Taken together, AI summaries from Google, Bing and DuckDuckGo debunked the false narratives a majority of the time, but less successfully than standalone chatbots. They also failed to challenge false claims more often than conventional search results. Google’s AI Overview rejected most of the narratives it encountered, while Bing’s summaries failed on most of the relevant tests; DuckDuckGo fell between the two.
Microsoft told NPR that its AI responses were grounded in search results and encouraged users to inspect sources. Google disputed parts of the methodology, arguing that some responses classed as failures still supplied useful context and that the prompts represented unusual rather than typical searches.
Chatbots need primary sources, not just the internet
The larger problem is not simply hallucination. NPR found that state-controlled and state-aligned outlets appeared among chatbot citations at broadly similar rates to conventional search links. For Claude specifically, such sources appeared more often in answers that failed to debunk a narrative than in answers that succeeded. Giving a model internet access therefore does not remove the weaknesses of the underlying information environment.
Academic evidence points to the same distinction. A 2026 ACL study tested 15 models against more than 6,000 PolitiFact claims and found that adding web search produced only moderate improvements in fact-checking. Feeding the models curated PolitiFact material through a retrieval system improved performance far more dramatically.
A Nature study published in May helps explain why source quality matters. Researchers found evidence that state-controlled media can influence language-model training data and, ultimately, model output. Commercial models produced more favourable answers about Chinese political institutions when prompted in Chinese than when asked equivalent questions in English. NPR and NewsGuard conducted their experiment only in English, an important limitation on how broadly its results can be applied.
Citations themselves are not a guarantee either. An independent analysis of 98,020 factual claims generated by Google AI Overviews found that 11% were unsupported by the pages the system cited.
The NPR experiment therefore does not show that chatbots have replaced search, much less professional verification. It suggests something more specific: a system capable of synthesising and judging multiple sources can sometimes resist a propaganda narrative better than a ranked list of links. But that advantage remains dependent on the same resource influence operations are increasingly trying to manipulate — the web from which the system gathers its evidence.
[…] READ ALSOAgainst state propaganda, chatbots beat Google and Bing search results→ […]