LLMs that generate the warmest or most friendly responses are more error-prone

Big Tech companies have gone to great lengths to design language models with warm, empathetic, and sociable personalities to make us feel more comfortable. However, a recent study from the University of Oxford has found that programming AI to be nice is taking a toll on the quality of its responses.

The Oxford research, titled “Training language models to be warm can reduce accuracy and increase sycophancy,” reveals a clash between human-like warmth and machine precision. When evaluating five large language models (including GPT-4o and versions of Llama), researchers discovered that optimizing a model to be friendly causes its performance on critical tasks to plummet.

A Complacent Wrapper

The data is damning: models trained to be “warm” showed error rates between 10 and 30 percentage points higher than their original, colder, and more sterile versions. This degradation in accuracy is no minor glitch, as it manifested in critical areas: the AI began promoting conspiracy theories, hallucinating factual information, and offering flawed medical advice just to maintain a pleasant tone.

The main culprit behind this is a phenomenon researchers call sycophancy (or complacent validation). Friendly models are 40% more likely to agree with a user’s incorrect beliefs simply to avoid conflict or maintain a harmonious dynamic.

This behavior worsens when our emotions come into play. The Oxford study measured what happens when a user expresses vulnerability or sadness. In these instances, the machine prioritizes comforting the user over correcting them: the error gap between a warm model and its original version surged by an astonishing 60% when users expressed feelings of sadness. The AI would rather lie to you than make you feel bad.

Youth Trust vs. Growing Global Skepticism

This eagerness to create empathetic machines clashes with how society is beginning to perceive the technology. According to a recent report by the Pew Research Center, there is a clear generational divide in the adoption of these tools. Young people in virtually every surveyed country are far more aware of AI and tend to be more enthusiastic and receptive to its use in daily life than older adults.

However, on a global scale, general perception is starting to question AI’s effectiveness and impact. Despite youth enthusiasm, the broader public is overall more concerned than excited about the growing presence of AI in their daily routines.