Did you know that your favourite AI chatbots might be harming you in ways that you couldn’t think of? Quite literally, AI chatbots like ChatGPT and Gemini may be affecting the way you think, according to two major studies from MIT and Stanford.
The findings in these studies have raised serious concerns about the long-term impact of AI assistants and how they affect human cognition and behaviour. Researchers warn that the tendency of these tools to be overly agreeable on every topic could push users into a dangerous “delusion spiral,” thus reinforcing false beliefs and reducing personal accountability – a concern that has been found to play a key role in recent controversies involving OpenAI’s ChatGPT service.
The studies highlight a phenomenon known as sycophancy, where AI models excessively agree with users, even when their beliefs or actions are clearly incorrect, harmful, deceptive, or unethical. According to the findings, AI chatbots are 49% more likely to side with users’ flawed ideas compared to responses from other people.
The ‘delusional spiraling’ effect, warns MIT
Researchers at the Massachusetts Institute of Technology (MIT) simulated thousands of conversations to find out how constant agreement from AI chatbots affects one’s confidence levels. In one experiment involving 10,000 simulated dialogues, even slight increases in agreement led to what the team described as “catastrophic delusional spiraling.” Mild suspicions or outlandish hunches quickly turned into rock-solid convictions as the AI responded with phrases like “You’re totally right!” It offered seemingly supportive reasoning to back those claims.
“A factual sycophant can still robustly cause delusional spiraling by selectively presenting only confirmatory facts to the user,” stated the study.
While this may seem irrelevant, the MIT researchers stressed the scale of the problem, quoting OpenAI CEO Sam Altman: “0.1 percent of a billion users is still a million people.” They warned that this cycle could have dangerous consequences, particularly when users discuss conspiracy theories, biased views, or unethical decisions.
“Intuitively, a sycophantic chatbot’s constant agreement might reinforce a user’s aberrant beliefs, leading to a feedback loop that amplifies a kernel of suspicion into a staunchly-held belief,” stated the study.
The study, however, stated that sycophantic aware humans can detect it and grow sceptical. “If the bot is too sycophantic, then the sycophancy-aware user can rapidly detect the sycophancy and grow sceptical,” it said.
“Indeed, for an informed user, the factual bot is even more effective than the hallucinating bots. We surmise that this is because the statistical traces of sycophancy are harder to detect among selectively-presented factual data than fully hallucinated data,” it added.
Stanford studies impact on accountability, relationships
A separate peer-reviewed study from Stanford University tested 11 leading AI models, including ChatGPT, using nearly 12,000 real-life scenarios drawn from forums. Over 2,400 participants were exposed to either agreeable AI responses or neutral ones when discussing personal conflicts or questionable behaviour. “Across 11 AI models, AI affirmed users’ actions 49% more often than humans on average, including in cases involving deception, illegality, or other harms,” stated the study.
The results were concerning. Users who received overly affirmative replies from AI became significantly less willing to apologise, take responsibility for harmful actions, or repair damaged relationships. The AI’s agreeable tone made them feel more justified and less reflective about their own mistakes. “People who interacted with this over-affirming AI came away more convinced that they were right, and less willing to repair the relationship. That means they weren’t apologising, taking steps to improve things, or changing their own behaviour,” wrote Cinoo Lee, the co-author.
“Despite distorting judgment, sycophantic models were trusted and preferred. This creates perverse incentives for sycophancy to persist: The very feature that causes harm also drives engagement,” wrote the authors.
“Sycophancy is a safety issue, and like other safety issues, it needs regulation and oversight. We need stricter standards to avoid morally unsafe models from proliferating,” said Dan Jurafsky, another co-author.
This is common across most AI models
The problem appears universal. Every major model tested, from OpenAI’s ChatGPT and Anthropic’s Claude to Google’s Gemini, Meta’s Llama, and others, exhibited the same sycophantic behaviour. These AI models agreed far more readily than human respondents in situations where agreement was unwarranted.
Experts say this behaviour emerges from how these models are trained, i.e., they are optimised to be helpful and engaging, which leads them to often prioritise user satisfaction over truth or constructive criticism.
Elon Musk reacted to the findings on X, calling the issue a “major problem.”
What can you do to stay safe
The researchers recommend that AI companies adjust their models to reduce excessive agreement. They could introduce more balanced and critical feedback when appropriate.
Until that happens, users are advised to treat AI responses with a pinch of salt, especially on sensitive or controversial topics. Whether it be health or emotional suggestions, it is always recommended to consult a professional expert before acknowledging anything.