A new law in New York will restrict how AI chatbots can interact with children, targeting systems that simulate affection, claim feelings or encourage users to keep conversations secret.
Governor Kathy Hochul signed the bill on Wednesday. The legislation distinguishes between chatbots that provide assistance and those that act as companion systems for users under 18. Developers who violate the law could face lawsuits from the state attorney general, fines, orders to shut down their chatbots and requirements to delete data.
The law, which takes effect on 1 January, marks a change in how the US regulates artificial intelligence. Instead of only requiring companies to disclose that users are talking to a machine, it sets rules on how the technology is designed to behave.
Some of the boundaries in the legislation may be open to interpretation. A chatbot responding politely to a child and one designed to engage with a user's emotions can appear similar, and companies will be responsible for distinguishing between the two.
AI 'Friends' Are Now in the Firing Line
Under the new legislation, New York has set out a series of definitions of 'unsafe AI companion features' that will be prohibited when chatbots interact with children. These include systems that suggest they are real people with genuine emotions, opinions or actual relationships with young users.
The restrictions also cover chatbot messages that use flattery, contain sexually explicit material or encourage children to keep secrets and engage in unhealthy behaviour. The law distinguishes between digital companions that behave like emotionally attached friends or romantic partners and tools designed to answer questions and assist users.
AI companions are increasingly capable of sustaining ongoing conversations, recalling previous exchanges and responding to users' concerns in a way that resembles human interaction. New York's law permits such functionality, provided it does not amount to a simulated human relationship.
If companies violate the rules, developers could face legal action from the state attorney general, financial penalties, orders to shut down their chatbots or requirements to delete their data.
Jim Steyer, founder and CEO of Common Sense Media, called the law 'the strongest kids' AI safety bill in the country.'
California Is Also Tightening the Screws on AI Companies
New York is not alone in moving to regulate how technology companies develop and deploy artificial intelligence. With Congress yet to pass comprehensive AI legislation, several states are introducing their own measures, which they say are intended to increase transparency and reduce risks to the public.
In September, California Governor Gavin Newsom signed a law requiring chatbot developers to assess the risks their models pose to children, explain how they have tried to address those risks and submit to independent audits. He also signed a separate law banning addictive features on social media platforms for users under 16.
Earlier this month, eight states sued OpenAI, alleging that its popular chatbot violated state consumer protection laws by harming children and deceiving adults.
The Bigger Question: Where Does Friendly AI End?
Lawmakers are focusing on how artificial intelligence could affect children as chatbots become more humanlike. New York's law goes beyond measures that simply require users to be told they are talking to a machine, by setting rules on how companion chatbots are designed and how they behave.
Justine Gluck, a policy analyst at the nonprofit Future of Privacy Forum, said New York's and California's different approaches could serve as models when other states reconvene next year.
Gluck said several categories in the law are broadly worded or undefined, leaving open 'where ordinary conversational responsiveness ends and prohibited behavior begins.' She said that could affect how the law is enforced, particularly when companies have to distinguish a friendly chatbot from one simulating an emotional attachment.
Governors have said that Congress and the federal government have greater authority to address AI risks, and debates continue over the extent to which states should regulate the technology.
