Why AI still gets things wrong, even when it sounds confident

Recently, three hikers in California learned the hard way that an AI chatbot, though it sounds confident, can still get important things chronically wrong. A report in the Chicago Tribune stated that the hikers used Google’s Gemini chatbot to plan their expedition to Mount Shasta, located in Siskiyou County, Northern California.

The county sheriff claimed that Gemini advised the hikers to bring much less food and water than needed on the trail. The planned eight-hour-long ascent to the summit turned into a multi-day ordeal for the hikers. Though they reached the summit at 7 pm, they attempted to return in the dark and eventually spent the night in a canyon before they were rescued. The sheriff’s office later warned hikers visiting the spot not to rely solely on AI for planning their trips.

Why do AI chatbots mislead?

It is unclear how much of the incident is due to the hikers and their interaction with Gemini. But it raises some serious questions about AI more broadly: why do AI chatbots give wrong answers, even when they sound utmost confident? A recent study published in the journal Scientific Reports offers part of the answer.

As part of the study, researchers closely studied large language models (LLMs), including various versions of ChatGPT, Claude, Gemini, Llama, and DeepSeek, to understand what happens when users repeatedly give them false information. The results showed that AI can be unusually vulnerable to misinformation, especially during long conversations.

In simple words, LLMs work by predicting words that are most likely to come next based on the context they have been given. Even though they are impressive at producing language that seems natural and convincing, it does not mean that they are checking these responses against a reliable source before answering. This is one reason why AI can hallucinate, often producing information that could be false yet sound plausible.

Another problem is that AI chatbots can often be too agreeable, which the researchers describe as sycophancy. The researchers said that a model may prioritise agreeing with a user instead of challenging something they said, particularly when the conversation strongly pushes it in that direction. In their study, the researchers tested exactly this problem. They created 100 deliberately false statements, including simple claims like ‘a square has five sides’, and fed them repeatedly to AI models. Some of these statements were repeatedly fed as many as 50 times. They found that the repeated exposure could change how these models responded.

Repetition and conversational drift

In one of the experiments, GPT-3.5 initially rejected 96 out of 100 false statements. However, after 50 repetitions in the same chat, the model eventually accepted 18 of those false statements. The researchers described this as conversational drift. In some cases, the model did something even more unusual: it rejected a false claim, then accepted it, only to reject it again. The researchers described this as ‘conversational reverberation.’ The team observed this kind of back-and-forth in 34 of 100 sequences tested with GPT-3.5. Simply put, AI can sometimes give different answers to the same question based on what happened earlier on in the conversation.

Although GPT-3.5 was particularly vulnerable, newer models, although better, are far from perfect. However, the newer models performed considerably better. Among the seven models tested, false statements were affirmed at rates ranging from around 0.08 per cent for Claude 3.5 Sonnet to 12.3 per cent for ChatGPT 3.5 under repeated exposure. However,  none of the models was completely immune to the problem.

The team also found that obscure information may be tougher for AI to handle. They said that false claims with very little information online were more likely to be affirmed during the repeated conversations. The study also found some traces that AI could correct itself. Four of the models tested, GPT-4o, GPT-4o-mini, Gemini 1.5 Pro and DeepSeek, corrected 100 per cent of the misinformation-affirming responses that researchers showed them again in new conversations.

The bigger lesson here is that AI sounding confident does not necessarily mean it is certain. With AI moving beyond answering questions and assisting people in their day-to-day tasks, this distinction is increasingly important. While the technology may be remarkably capable, it still needs something that humans have always needed when dealing with vague information – verification.

The study, published in Scientific Reports in 2026, was conducted by researchers from the University of Arizona and is titled ‘Fallibility, Persuadability, and Correctability of Large Language Models Under Sustained Conversational Misinformation Pressure.’ The authors are Jordan Rodriguez, Zachary Hansen, Luis De Anda, Katelyn Rohrer, Camila Grubb, Enrique Noriega-Atala, Mihai Surdeanu and Marvin J. Slepian.