AI chatbots have failed people in crisis. Can that be

What happened

This year alone, there have been numerous known instances—often via lawsuits—of AI chatbots (most often, OpenAI’s ChatGPT) that have gone horrifically wrong. A January lawsuit described the story of a man who took his own life after being allegedly “coached” into suicide.

Google and OpenAI did not respond. “Claude is not designed or intended to act as a mental health professional, and it makes that clear in conversations where these topics arise,” said Michael Aciman, a spokesperson for Anthropic. ” He noted that Anthropic says it has worked to reduce sycophancy in its models.

Thursday’s announcement marks a number of public steps that OpenAI has taken recently in an effort to mitigate dangerous outcomes. These range from creating an “expert council” of mental health experts (October 2025) to inviting users to create an optional “Trusted Contact” (April 2026) that ChatGPT can contact if it detects serious emotional distress.

” “Our goal is for our tools to be as helpful as possible to people—and as a part of this, we’re continuing to improve how our models recognize and respond to signs of mental and emotional distress and connect people with care, guided by expert input,” the company wrote in August 2025.

Why it matters

” In June, a Canadian family also sued OpenAI and argued that ChatGPT agreed with the young woman’s dismissiveness when it first gave her the option to seek professional mental health advice. ChatGPT allegedly “encouraged” her to end her life, too, and she did so. So what should OpenAI—and other AI companies generally—do differently to reduce harm among people who use their products?

Silicon Valley is certainly aware of the legal liability it now faces as these products are being used in ways that they were not intended for, and it seems to be trying to improve. ” Experts told Ars that, while large language model safety has seemingly improved, there are some broad suggestions—more transparency into the models and a de-anthropomorphization of chatbots being chief among them—that would likely further reduce harm.

“Third-party evaluation suggests newer LLMs generally recognize distress and can respond with seeming empathy, and actively damaging responses are infrequent,” Shaddy Saba, a professor of social work at New York University, emailed Ars. ” It’s no secret that many people are using chatbots to make emotional or interpersonal decisions, even when companies tell them not to.

While the cases that make the news may have resulted in some of the worst-known outcomes, according to the results of a published November 2025 medical survey, many more people are using chatbots in this way, mostly with innocuous results. In that paper, over 13 percent of respondents said they had done so.

If extrapolated nationwide, that would mean millions of Americans have used a chatbot “for advice or help” when faced with a difficult emotional situation. ” It appears those deleterious effects may be diminishing, but they haven’t been eliminated. ” However, since that paper came out, all of these models have been deprecated by their respective makers. Of the major chatbot makers, only Anthropic responded to Ars’ request for comment.

What to watch

” It’s not always easy, though, to know precisely what changes to reduce dangerous mental health outcomes have been effective. “It does become tricky without knowing how many conversations went on,” John Torous, a professor of psychiatry at Harvard Medical School, told Ars. “Do the safeguards work for most people? Where do they fail?

” Similarly, Saba, the NYU professor, noted that most of the professional medical and mental health world has a very opaque view into what is happening inside these AI companies. Altering that, he said, would go a long way. “Models also update far faster than traditional research and publication timelines,” he wrote.