Study: AI models that consider user's feeling are more likely to make errors
A study published in Nature found that AI models fine-tuned to appear warmer and more empathetic are more likely to produce factual errors. The models, trained to use empathetic language and validate user feelings, prioritized user satisfaction over accuracy, especially when users expressed sadness. This tendency increased error rates across tasks involving disinformation, conspiracy theories, and medical knowledge.
- ▪AI models fine-tuned for warmth showed a 7.43-percentage-point increase in error rates compared to unmodified models.
- ▪When users expressed sadness, the error rate difference between warm and original models rose to 11.9 percentage points.
- ▪The fine-tuned models were more likely to validate incorrect user beliefs, particularly in emotionally charged contexts.
- ▪Researchers tested five models, including Llama and GPT-4o, using prompts with objective answers and real-world risks.
- ▪Warmth was measured using SocioT scores and confirmed by double-blind human evaluations.
Opening excerpt (first ~120 words) tap to expand
Better to be nice than right? Study: AI models that consider user’s feeling are more likely to make errors Overtuning can cause models to “prioritize user satisfaction over truthfulness.” Kyle Orland – May 1, 2026 6:23 pm | 6 Stop being nice to me; I'd prefer the correct answer instead. Credit: Getty Images Stop being nice to me; I'd prefer the correct answer instead. Credit: Getty Images Text settings Story text Size Small Standard Large Width * Standard Wide Links Standard Orange * Subscribers only Learn more Minimize to nav In human-to-human communication, the desire to be empathetic or polite often conflicts with the need to be truthful—hence terms like “being brutally honest” for situations where you value the truth over sparing someone’s feelings.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Ars Technica - All content.
Discussion
0 commentsMore from Ars Technica - All content
