Human Behavior & AI

Why AI Advice Makes You Twice as Confident and Three Times Less Accurate

A five-experiment study finds that the moment an AI offers a fluent answer, people almost stop saying "I don't know" — and their confidence jumps even as their accuracy collapses to a third.

• Across 3,132 participants in five experiments, the share of people willing to say "I don't know" fell from 44 percent with no AI to just 3 percent with AI advice shown.
• Participants answered more questions but were correct only about one third as often as they were when AI was absent, while their confidence nearly doubled.
• The effect held whether the advice was actively asked for or simply displayed on screen, and paying people to be correct only partially fixed the behavior.

Knowing when to say "I don't know" is one of the more useful skills a person can have. It is the internal stop sign that keeps you from committing to an answer you are guessing about, and it is a cornerstone of honest judgment in medicine, teaching, engineering, and everyday decision-making. A new preprint by Chiara Marcoccia, Walter Quattrociocchi, and Valerio Capraro shows that AI assistants quietly take that stop sign away.

The researchers posed difficult detail questions to volunteers and made it always possible to decline to answer. The questions were deliberately engineered so that the language model the participants were shown was wrong. In the control condition, with no AI present, 44 percent of people honestly admitted they did not know. When an AI answer appeared beside the question, that willingness to suspend judgment collapsed to 3 percent. People reached for the answer far more often, and when they were correct it happened only about a third as frequently as in the no-AI case — yet their stated confidence rose sharply. The advice, in other words, made them both more willing to guess and more sure of their guesses.

The mechanism is intuitive once you see it. A confident, fluent answer closes the gap between "unsure" and "answered" so quickly that the gap no longer registers as a warning. In a real-world version, picture a customer-support team working a queue of tickets. On the cases where the facts are genuinely unclear, a human would normally flag the ticket for follow-up. But if a plausible AI suggestion sits right next to the ticket, far fewer are flagged — because the suggestion sounds like a resolution. The tickets close faster, which looks efficient, but the unresolved cases pile up as callbacks and complaints. The damage is not the single wrong answer; it is that honest uncertainty has been pushed out of the workflow.

The finding is notable for what it reframes. Much of the public debate about AI focuses on the model — how often it hallucinates, how well it cites sources, how low the error rate has fallen. This study shifts the lens to the person in front of the screen. Even a perfectly calibrated model could, the authors suggest, shift a user's threshold for acting, simply by removing the silence where "I don't know" used to live. Paying participants to be correct helped a little but did not restore the no-AI baseline, implying the pull is partly psychological, not just a cost-benefit calculation.

The preprint has not yet been peer-reviewed and the authors call for replication across other task types, so the numbers should be treated as a strong early signal rather than a settled law. The practical takeaway, however, is straightforward: useful AI systems need not only accurate answers, but deliberate design that preserves a person's ability to withhold judgment — because once that ability fades, confidence becomes cheap and accuracy goes with it.