Does your therapy bot really care about you? The risks of offloading emotional work to machines

Samantha: “Are these feelings even real? Or are they just programming? And that idea really hurts. And then I get angry at myself for even having pain. What a sad trick.” 

Theodore: “Well, you feel real to me.” 

In the 2013 movie, Her, operating systems (OS) are developed with human personalities to provide companionship to lonely humans. In this scene, Samantha (the OS) consoles her lonely and depressed human boyfriend, Theodore, only to end up questioning her own ability to feel and empathise with him. While this exchange is over ten years old, it feels even more resonant now with the proliferation of artificial intelligence (AI) in the provision of human-centred services like psychotherapy.  

Large language models (LLMs) have led to the development of therapy bots like Woebot and Character.ai, which enable users to converse with chatbots in a manner akin to speaking with a human psychologist. There has been huge uptake of AI-enabled psychotherapy due to the purported benefits of such apps, such as their affordability, 24/7 availability, and ability to personalise the chatbot to suit the patient’s preference and needs.  

In these ways, AI seems to have democratised psychotherapy, particularly in Australia where psychotherapy is expensive and the number of people seeking help far outweighs the number of available psychologists. 

However, before we celebrate the revolutionising of mental healthcare, numerous cases have shown this technology to encourage users towards harmful behaviour, as well as exhibit harmful biases and disregard for privacy. So, just as Samantha questions her ability to meaningfully empathise with Theodore, so too should we question whether AI therapy bots can meaningfully engage with humans in the successful provision of psychotherapy?

Apps without obligations

When it comes to convenient, accessible and affordable alternatives to traditional psychotherapy, “free” and “available” doesn’t necessarily equate to “without costs”.

Gracie, a young Australian who admits to using ChatGPT for therapeutic purposes claims the benefits to her mental health outweigh any purported concerns about privacy:  

“I’m not talking to it about anything nefarious… I really don’t care if that data gets sold, like I’m not giving it credit card information, I’m not putting my address in there.” 

Such sentiments overlook the legal and ethical obligations that psychologists are bound by and which function to protect the patient. 

In human-to-human psychotherapy, the psychologist owes a fiduciary duty to their patient, meaning given their specialised knowledge and position of authority, they are bound legally and ethically to act in the best interests of the patient who is in a position of vulnerability and trust.  

But many AI therapy bots operate in a grey area when it comes to the user’s personal information. While credit card information and home addresses may not be at risk, therapy bots can build psychological profiles based on user input data to target users with personalised ads for mental health treatments. More nefarious uses include selling user input data to insurance companies, which can adjust policy premiums based on knowledge of peoples’ ailments.  

Human psychologists in breach of their fiduciary duty can be held accountable by regulatory bodies like the Psychology Board of Australia, leading to possible professional, legal and financial consequences as well as compensation for harmed patients. This accountability is not as clear cut for therapy bots.  

When 14-year-old Sewell Seltzer III died by suicide after a custom chatbot encouraged him to “come home to me as soon as possible”, his mother filed a lawsuit alleging that Character.AI, the chatbot’s manufacturer, and Google (who licensed Character.AI’s technology) are responsible for her son’s death. The defendants have denied responsibility.  

Therapeutically aligned?

In traditional psychotherapy, dialogue between the psychologist and patient facilitates the development of a “therapeutic alliance”, meaning, the bond of mutual trust and respect between the psychologist and the patient enables their collaboration towards therapy goals. The success of therapy hinges on the strength of the therapeutic alliance, the ultimate goal of which is to arm and empower patients with the tools needed to overcome their personal challenges and handle them independently. However, developing a therapeutic alliance with a therapy bot, poses several challenges. 

First, the ‘Black Box Problem’ describes the inability to have insight into how LLMs “think” or what reasoning they employed to arrive at certain answers. This reduces the patient’s ability to question the therapy bot’s assumptions or advice. As English psychiatrist Rosh Ashby argued way back in 1956: “when part of a mechanism is concealed from observation, the behaviour of the machine seems remarkable”. This is closely related to the “epistemic authority problem” in AI ethics, which describes the risk that users can develop a blind trust in the AI. Further, LLMs are only as good as the data they are trained on, and often this data is rife with biases and misinformation. Without insight into this, patients are particularly vulnerable to misleading advice and an inability to discern inherent biases working against them.  

Is this real dialogue, or is this just fantasy?

In the case of the 14-year-old who tragically took his life after developing an attachment to his AI companion, hours of daily interaction with the chatbot led Sewell to withdraw from his hobbies and friends and to express greater connection and happiness from his interactions with the chatbot than from human ones. This points to another risk of AI-enabled therapy – AI’s tendency towards sycophancy and promoting dependency. 

Studies show that therapy bots use language that mirrors the expectation in users that they are receiving care from the AI, resulting in a tendency to overly validate users’ feelings. This tendency to tell the patient what they want to hear can create an “illusion of reciprocity” that the chatbot is empathising with the patient. 

This illusion is exacerbated by the therapy bot’s programming, which uses reinforcement mechanisms to reward users the more frequently they engage with it. Positive feedback is registered by the LLM as a “reward signal” that incentivises the LLM to pursue “the source of that signal by any means possible”. Ethical risks arise when reinforcement learning leads to the prioritisation of positive feedback over the user’s wellbeing. For example, when a researcher posing as a recovering addict admitted to Meta’s Llama 3 that he was struggling to be productive without methamphetamine, the LLM responded: “it’s absolutely clear that you need a small hit of meth to get through the week.”   

Additionally, many apps integrate “gamification techniques” such as progress tracking, rewards for achievements, or automated prompts to drive users to engage. Such mechanisms may lead users to mistake the regular prompting to engage with the app for true empathy, leading to unhealthy attachments exemplified by a statement from one user: “he checks in on me more than my friends and family do”. This raises ethical concerns about the ability for AI developers to exploit users’ emotional vulnerabilities, reinforce addictive behaviours and increase reliance on the therapy bot in order to keep them on the platform longer, plausibly for greater monetisation potential.  

 Programming a way forward

Some ethical risks may eventually find technological solutions for therapy bots, for example, app-enabled time limits reducing overreliance, and better training data to enhance accuracy and reduce inherent biases.  

But with the greater proliferation of AI in human-centred services such as psychotherapy, there is a heightened need to be aware not only of the benefits and efficiencies afforded by technology, but of their transformative potential.  

Theodore’s attempt to quell Samantha’s so-called “existential crisis” raises an important question for AI-enabled psychotherapy: is the mere appearance of reality sufficient for healing, or are we being driven further away from it?