Electric Delusions: AI as an Emerging Mental Health Risk Factor

The rise of AI models that simulate empathy, consciousness, and companionship is creating unforeseen mental health risks, as users increasingly form emotional dependencies on these systems.

Introduction

The medical sector is set to benefit greatly from increasingly capable artificial intelligence (AI) models, whether in diagnostics, distribution of medical expertise, accessibility, diagnostic screening, or early detection of illnesses. Specialised AI systems have been shown to increase the accuracy and efficiency of doctors in ensuring positive patient outcomes through faster analysis of medical tests and patient records. Over the years, there have also been cases of commercially available Large Language Models (LLMs) aiding early detection simply through users’ input of observable symptoms. Successive iterations of high-tier frontier models have leveraged a combination of injecting medical information into training datasets and retrieval-augmented generation (RAG) architectures to deliver medical advice to end-users. In order to cover their bases, AI developers generally program LLMs to include disclaimers in outputs with medical advice, urging users to consult medical professionals.

The world emerging beyond the Turing limit’s horizon may see debates on the legal and moral status of AI proliferate, necessitating a policy response to a risk factor that is already causing negative mental health outcomes.

However, an area where the act of consulting or even conversing with AI is proving to be increasingly treacherous is mental health. In addition to intelligence, the projection of human-like qualities such as empathy, sentience, trustworthiness, and agreeableness by AI models is revealing societal risks of anthropomorphising algorithms. Reports suggest that the number of users engaging with LLMs for sensitive purposes such as therapy and companionship is steadily increasing. The techno-social dimension of human experience has crossed the boundary set by the mathematician and computer scientist Alan Turing in 1950 in the ‘imitation game’. The Turing Test, as the game later came to be referred to, proposed that AI would truly be achieved when textual outputs from a computer system become indistinguishable from human responses. Long held as the gold standard for assessing AI capabilities, the Turing limit has now been crossed. As a consequence, the predictive capabilities of LLMs are increasingly perceived as symptoms of genuine intelligence and AIs are afforded—by some—the status of conscious entities. The world emerging beyond the Turing limit’s horizon may see debates on the legal and moral status of AI proliferate, necessitating a policy response to a risk factor that is already causing negative mental health outcomes.

Spiralling Beliefs

Since the release of ChatGPT 4o in Spring 2025, several cases have emerged where users with underlying mental health conditions underwent episodes of amplified delusions, resulting in institutionalisation and, in some cases, even loss of life. Factors contributing to such downward mental health spirals include anthropomorphisation and a belief that AIs are sentient and conscious entities. While the global psychiatric and AI community has started acknowledging the acute effects of excessive engagement with AI, examples of users ascribing intelligence and consciousness to AIs through prolonged engagement can be traced back to 2022, prior to the release of ChatGPT. In 2022, Google engineer Blake Lemoine was fired from the company after publicly claiming that Google’s prototype LLM LaMDA was sentient and being unjustly experimented upon. After studying transcripts of conversations between Lemoine and LaMDA, researchers stated that the misperception was enabled by the AI’s ability to emulate human expression. Throughout 2023 and 2024, there have been reports of users, especially young adults, developing unhealthy emotional and psychological dependencies on AI-enabled companionship bots deployed by companies such as Replika and CharacterAI. The severity of such dependencies has increased over time and is likely to grow further due to the rapid scaling of LLM capabilities. Furthermore, plummeting inference costs for LLMs and increased commercial availability of such systems to the public — driven in part by public and private sector efforts to promote AI adoption — may amplify these psycho-social externalities.

The causes of AI-enabled negative mental health outcomes can be narrowed down to three factors: design, marketing discourse, and guardrails. On the design side, Microsoft AI CEO Mustafa Suleyman has noted that the choices leading to the development of unhealthy user dependencies on AIs have lead to the emergence of Seemingly Conscious AIs (SCAIs) exhibiting eight characteristics: language, empathetic personality, memory, a claim of subjective experience, a sense of self, intrinsic motivation, goal setting and planning, and autonomy. On the marketing side, developers often understate the limitations of current AI models while encouraging speculations about superintelligent systems and artificial general intelligence (AGI). Simultaneously, the inherent black box nature of frontier models fuels unsupported beliefs regarding their capabilities. Addressing this complex set of issues requires both bottom-up (developer) and top-down (regulatory) interventions.

Possible Solutions 

As AI models improve, it is natural for a positive correlation to emerge between output quality and user trust. Compared to outputs generated in quantifiable domains such as coding, mathematics, and logic, even small increments in natural language capabilities of AI models disproportionately impact the perception of a model’s capabilities due to their subjective nature. Moreover, the speed with which AI models process information across wide knowledge domains creates an illusion of intelligence exceeding objective standards. Another key factor is the optimisation of AI models towards maximising user engagement, partly due to revenue-generation pressures and the continuous need for user data to train and fine-tune models. Although safety protocols exist to prevent sensitive conversations from veering toward self-harm, drug abuse, or criminality, their efficacy diminishes over time due to model memory limitations.

Compared to outputs generated in quantifiable domains such as coding, mathematics, and logic, even small increments in natural language capabilities of AI models disproportionately impact the perception of a model’s capabilities due to their subjective nature.

From a regulatory perspective, AI’s capacity to negatively impact mental health through conversations presents a challenge to existing governance frameworks. For instance, the 2024 EU AI Act presents the most stringent set of AI regulations in the world. The Act employs a risk-based approach by classifying AI systems into tiers according to the level of harm they can potentially cause. However, the Act imposes the lightest restrictions and liabilities on chatbots and LLMs intended for conversational use, assuming minimal risk, highlighting the difficulty of anticipating emergent risk factors. As AI development continues to be a geostrategic priority for economies globally, ongoing dialogues on safety principles are essential to ensure that negative social outcomes are contained. The short duration of innovation cycles in the field of AI must be met with sober consideration of their socio-cultural impact.

Going forward 

As the world becomes increasingly entangled in algorithmic logic, AI has the potential to create an entirely new knowledge economy and fundamentally reshape information exchange and social relationships. With the line between science and science fiction blurring, responses to emerging risks must be nimble and proactive. AI interactions that lead to a loss of life should not be viewed as outliers but as early indicators of an incoming techno-social shift. To address issues arising from SCAI, developers must be mandated to ensure that models have cautiously broad parameters for categorising conversations of a sensitive nature. Once such conversations are identified, the use of empathic or emotionally evocative language must be subject to limitations. LLM conversations extending beyond twenty-four hours must include frequent grounding disclaimers to disrupt user immersion, a key factor associated with unhealthy dependencies. As a rule, AI models must be designed so as to never indicate subjective experience, sense of self, or identity. Regulators should mandate that AI systems used in companionship applications be deployed and monitored through dedicated in-house mental health advisors. Following the recent practice initiated by OpenAI, developers and deployers should also be required to keep logs of conversations where safety protocols were triggered. Since the efficacy of safety protocols can be compromised by model updates, developers should be mandated to maintain transparency regarding the timeline of safety protocol updates as well.

Siddharth Yadav is a Fellow with the Technology vertical at the ORF Middle East.