Eating Disorder Helpline Takes Down Chatbot After Its Advice Goes Horribly Wrong

Maxwell told the Daily Dot that the bot even tried to get her to track her calorie intake and weigh herself constantly. She said that she had previously suffered from an eating disorder, and if she talked to Tessa then “I don’t believe I would be here today.” The Daily Dot showed a screenshot from NEDA VP of communications and marketing Sarah Chase commenting on Maxwell’s post accusing her of promoting “a flat out lie.” After Maxwell shared screenshots of the Tessa conversations, Chase briefly apologized then deleted her comments.

Chase previously told us that the chatbot “can’t go off script,” and was only supposed to walk users through an eating disorder prevention program and link to other resources on NEDA’s website.

The nonprofit’s CEO Liz Thompson told Gizmodo:

“With regard to the weight loss and calorie limiting feedback issued in a chat recently, we are concerned and are working with the technology team and the research team to investigate this further; that language is against our policies and core beliefs as an eating disorder organization. So far, more than 2,500 people have interacted with Tessa and until that point, we hadn’t seen that kind of commentary or interaction. We’ve taken the program down temporarily until we can understand and fix the “bug” and “triggers” for that commentary. “

Thompson added that Tessa isn’t supposed to be a substitute for in-person mental health care and that those in crisis should text the crisis text line.

A paper from 2022 about the eating disorders chatbot describes a study sample size of 2,409 who used the ED chatbot after seeing ads on social media. The study authors said they reviewed more than 52,000 comments from users to “identify inappropriate responses that negatively impacted users’ experience and technical glitches.” Researchers noted the biggest issue with the chatbot was how limited it was responding to “unanticipated user responses.”

Though as Maxwell pointed out in a follow up post, outsiders have no way to tell how many of those 2,500 people received the potentially harmful chatbot commentary.

Other professionals tried out the chatbot before it was taken down. Psychologist Alexis Conason posted screenshots of the chatbot to her Instagram showing the chatbot provided the same “healthy and sustainable” weight loss language as it did to Maxwell. Conason wrote that the chatbot’s responses would “further promote the eating disorder.”

What’s even more confusing about the situation is how NEDA seems hell bent on claiming Tessa isn’t AI, but some much more blasé call and response chatbot. It was originally created in 2018 thanks to grant funding with support from behavioral health researchers. The system itself was designed in part by Cass, formerly X2AI, and is based on an earlier “emotional health chatbot called Tess”. Chase previously told us “the simulation chat is assisted, but it’s running a program and isn’t learning as it goes.”

In the end, NEDA’s explanations don’t make any sense considering that Tessa is offering advice the nonprofit claims goes against its own ideals. A paper from 2019 describes Tess as an AI based on machine learning and “emotion algorithms.”

Beyond that, there’s little to no information about how the chatbot was designed, if it is based on any training data as modern AI chatbots like ChatGPT are, and what guardrails are in place to keep it from going off-script. Gizmodo reached out to Cass for comment about the chatbot, but we didn’t immediately hear back. The company’s page describing Tessa has been removed, though the page was active as recently as May 10, according to the Wayback Machine.

Services Marketplace – Listings, Bookings & Reviews

Entertainment blogs & Forums