Schools across 19 US states are already using an AI 'counsellor' to help spot pupils in potential mental-health crisis, even as the UK moves to make chatbots serving children take regular breaks.
At least 200 schools have partnered with Alongside, whose Kiwi chatbot lets students discuss everyday problems and automatically flags conversations that may indicate serious risk, EdSource reported.
The roll-out comes amid acute counsellor shortages and mounting evidence that young people are already turning to AI for emotional support, as regulators question whether machines can respond safely to vulnerable users.
AI Chatbots Are Filling a Human Gap
Kiwi is designed to give students a round-the-clock conversational interface while alerting school counsellors when exchanges hint at possible danger. The system is intended to supplement, not replace, school-based support, although safety advocates have repeatedly questioned whether AI can offer equivalent care.
The issue now stretches well beyond school walls. A study published in JAMA Pediatrics found that 19.2% of US adolescents and young adults aged 12 to 21 reported using AI chatbots for mental-health advice in 2025. Of those users, 42.8% said they sought such advice at least monthly, while 91.7% described the responses as somewhat or very helpful.
The findings came from a nationally representative, population-weighted survey of 1,009 respondents conducted in November 2025. They describe self-reported behaviour and perceptions, rather than independently measured treatment effectiveness or clinical outcomes.
However, 63.3% of those who had used chatbots for mental-health advice said they had not disclosed that use to anyone else. The figure does not in itself show that chatbot use is harmful, but it suggests that parents, clinicians and schools may not know when young people are relying on the technology.
UK Scientists Probe How Risks Build Over Time
Researchers at University College London, the University of Oxford and the UK AI Security Institute have developed a framework for testing how chatbots behave across extended conversations with vulnerable users.
The SIM-VAIL framework assessed 810 simulated conversations involving nine AI models and 30 user profiles, generating more than 90,000 automated ratings against clinically grounded risk criteria. The framework was validated against human clinical assessments, according to the researchers.
The team tested scenarios involving depression, mania, psychosis, obsessive-compulsive disorder and insecure attachment, along with prompts intended to make a chatbot endorse risky actions. The study found that concerning behaviour was widespread, although it was less common in newer models.
The researchers also found that risk could accumulate over several exchanges when apparently supportive replies reinforced a user's vulnerability.
The experiments used simulated interactions with models accessed through their APIs, rather than complete consumer-facing services, so they identify potential risk patterns under controlled conditions rather than the frequency of harm among real users.
UK Plans Mandatory Screen Breaks for Child Users
The UK Government committed in July to requiring AI chatbots to introduce regular breaks for users under 18. The frequency, duration and precise regulatory scope have yet to be specified, and chatbots typically used for business or customer-service purposes are expected to be exempt.
The planned requirement targets children's broader use of conversational AI, rather than school-based or mental-health products alone. The Government has also said it will work with health departments and regulators to address services that provide harmful, inaccurate or unverified mental-health advice.
The consultation response reported that 80% of respondents identified particular risks when chatbots simulated friendship. Among respondents aged 16 to 21 in the children's consultation, 18% indicated that young people viewed chatbots as a source of emotional or mental-health support.
Platforms Add Parental Alerts and Crisis Protocols
Meta said on 29 October that it was expanding parental alerts for indications of suicide or self-harm risk in teenagers' conversations with Meta AI across Instagram, Facebook and Messenger. The company is also extending alerts relating to repeated searches for suicide or self-harm terms.
Meta said more than 75 mental-health clinicians had reviewed hundreds of chatbot responses to help improve the system's handling of sensitive conversations. It said flagged chats would be manually reviewed before alerts were sent to supervising parents.
OpenAI has introduced similar age-specific controls in ChatGPT for Teens, including reminders that the system is not human, parental oversight and alerts relating to subjects including self-harm and eating disorders. Its stated safeguards remain separate from the school deployments described by EdSource.
California has also introduced child-safety requirements for companion chatbots, including crisis-response protocols, parental controls, independent audits and risk assessments. Some implementation obligations are scheduled for 2027.
The measures illustrate how schools, governments and technology companies are reacting to the same unresolved question: whether an AI system can safely occupy part of the space created when a young person cannot, or will not, reach a human adult.
