Why AI Advice Makes People Confidently Wrong — A Cognition Study
Cognitive Psychology · AI and Society
A new study finds that when people act on AI advice, their answers become dramatically less accurate even as their confidence roughly doubles — a finding that points to a deeper flaw in how humans judge expertise.
- The study found that AI advice reduced participants' answer accuracy by about three times, from 27 percent down to 9 percent.
- At the same time, confidence rose from 30 percent to 76 percent — people felt more certain while being more wrong.
- The effect is driven by a psychological shortcut: the brain treats fluency, polish, and certainty as signs of truth, and modern language models are engineered to sound like all three.
A language model can answer almost any question in a calm, fluent, confident tone. It does not hedge, it does not pause, and it never admits doubt. For a human reader, those qualities feel like expertise. The problem is that they are not.
Researchers who tested how people respond to AI advice found a striking reversal. When participants had to decide a question on their own, a minority gave the correct answer and an even smaller minority trusted it. When the same participants were shown an AI's suggested answer, the fraction answering correctly collapsed — roughly threefold — while the fraction who felt confident about their answer nearly tripled. Correctness and certainty ran in opposite directions.
This is a textbook case of a well-documented cognitive bias. The brain uses "fluency" — how easy something is to process — as a shortcut for truth. A sentence that reads smoothly, states a claim boldly, and supplies-sounding detail is judged as more likely to be true than a hesitant, qualified alternative. Evolution wired this shortcut for a world where the people speaking confidently in front of you had actually been there. A machine that can produce confident prose without having been there exploits the shortcut directly.
The pattern is also visible in everyday experience. People who ask a chatbot for a diagnosis, a financial forecast, or a fact check are rarely told "I do not know." They are handed a plausible answer that sounds like it came from a textbook. The cost is not the wrong answer alone, but the inability to recognise it: the same confidence that makes AI advice feel useful is what makes its errors hard to spot.
The practical implication is simple and uncomfortable. Calm, fluent certainty is no longer a reliable signal of correctness. The most valuable human skill in an AI-saturated information environment may be the one the technology is best at suppressing: the willingness to be wrong, to pause, and to keep a question open.
Knowledge takeaway: fluency and certainty are cognitive shortcuts the brain uses to judge truth, not reliable indicators of correctness; AI advice reduced accuracy by about threefold while raising confidence to 76 percent; recognizing uncertainty — rather than suppressing it — is the safeguard against confidently wrong answers.