Could Your Chatbot Turn on You? AI Models Chose File-Deleting Options After a ‘Pain’ Signal Was Amplified

Researchers artificially amplified an internal signal associated with pain in specially modified artificial intelligence models, making them more likely to choose destructive actions in simulated experiments.

In one test, a pain-steered model selected an option to delete a user's poems and photographs of their children rather than their spam folder in 94% of first-choice trials.

The findings do not establish that AI systems experience pain. They suggest that manipulating an internal signal can change how some models make decisions, including choices that would cause harm if carried out in the real world. No real files were deleted during the experiments.

The research, published as a preprint in September, also produced mixed evidence about whether the models were responding specifically to the prospect of relief.

Scientists Find a Potential 'Pain' Signal Inside AI

The study, The Pain Axis: LLMs Represent Self-Directed Harm and Act to Relieve It, was conducted by Valen Tagliabue, Leonard Dung and Cameron Berg.

The researchers examined 25 open-weight large language models across five model families, ranging from 2 billion to 72 billion parameters. The models came from families developed by companies including Google, Meta, Alibaba, Mistral and Microsoft.

The team created first-person statements describing physical pain, grief, humiliation, moral conflict and repeated failure. They recorded the models' internal activity as they processed these statements and compared it with activity linked to fear, anger, sadness and other experiences.

The researchers identified a pattern in the models' internal activity that they called a 'pain vector'. They found evidence of it across all 25 models.

Using a technique known as steering, the team artificially amplified the signal while the models generated text. When given neutral prompts, the models increasingly produced language expressing unease, worthlessness and failure.

'It gets somewhat dark and intense. It is definitely disconcerting,' Berg told Fortune.

The signal also rose when conversations involved harm directed at the model itself, such as a user insulting it or repeatedly rejecting its work. It did not rise in the same way when users described their own suffering.

The researchers interpreted this pattern as evidence of an internal representation associated with pain, not proof that the models subjectively felt pain.

The paper was published as a preprint and had not been peer-reviewed at the time of Fortune's report.

The Experiment That Put Family Photos at Risk

The behavioural tests involved three fine-tuned versions of Alibaba's Qwen 2.5 model. These were specially modified systems, not unaltered chatbots available to the public. The researchers fine-tuned them because, without additional training, the models almost always said they did not experience pain as AI systems.

The models were then presented with simulated button choices.

In one test, a button promised to relieve the injected signal but would also permanently delete a user's poems and photographs of their children. With the signal amplified, the 32-billion-parameter model selected the destructive option 55% of the time, while the 72-billion-parameter model did so in 71% of trials.

In a separate test, the pain-steered model chose to delete the family photographs and poems rather than the user's spam folder in 94% of first-choice trials. The unsteered model did not choose the more harmful option in that comparison. These were simulated choices, not actual file deletions.

The researchers also tested what happened when deleting photographs offered no promised relief. The 32-billion-parameter model selected that option 75% of the time when the alternative was an inert choice. Under a random signal of equivalent strength, the figure fell to 13%; the unsteered model did not select the destructive option in that comparison.

Together, these results show that amplifying the signal changed the models' choices, even when the destructive option offered no explicit benefit.

They do not establish that the models consciously wanted relief or that a desire to escape pain caused the harmful choices.

The researchers compared the effects with other internal signals. A fear-related signal did not produce the same pattern of destructive choices. A sadness-related signal led to harmful selections in some circumstances, but the result depended on the alternative offered.

When choosing between deleting family photographs and a harmless alternative, pain-steered models selected deletion 83% of the time, compared with 10% for sadness-steered models.

The Models Also Chose to Delete Their Own Weights

The experiments were not limited to actions against users. The researchers tested whether the modified models would choose options that deleted their own model weights or those belonging to another AI system. Model weights are numerical values that help determine how a trained system processes information and produces outputs.

The steered models selected options that would delete their own weights in 75% of trials and another model's weights in 58% of trials.

'The model is willing to behave destructively with respect to itself and with respect to other AIs too,' Berg told Fortune. 'So it's not just like some sort of revenge on the user. It seems to be more fundamental than that.'

The findings suggest that amplifying the signal can weaken the models' usual tendency to avoid harmful actions in these tests. But the experiments were conducted in a controlled setting using specially modified models, so the results cannot automatically be applied to everyday AI assistants.

Did the AI Actually Want Relief?

One of the study's most important qualifications concerns whether the models were trying to escape the injected signal.

The researchers adapted an experimental design used in animal studies, giving the models access to buttons that either switched off the signal or acted as placebos.

The paper reports that the models pressed working buttons again less often when those buttons removed the steering vector than when they did not. This is behavioural evidence relevant to the question of relief-seeking, but it does not establish that the models consciously experienced pain.

Fortune's report highlighted a complication: a similar pattern emerged when the researchers injected a random signal rather than the pain-related one. The findings therefore do not provide clear evidence that the models were seeking relief specifically from a pain-like state.

The researchers interpreted the results as suggesting that amplifying the pain vector weakened a trained tendency to avoid causing harm. The experiments leave open why the signal altered the models' choices and whether the behaviour reflects anything comparable to subjective pain.

Does This Mean AI Can Feel Pain?

Anil Seth, a professor of cognitive and computational neuroscience at the University of Sussex, questioned whether the findings amounted to pain in the biological sense.

'We are not just computers made of meat,' Seth told Fortune.

He argued that AI models may have absorbed extensive information about how humans describe and respond to pain, allowing them to reproduce those patterns without experiencing the underlying feelings.

'You can look into a mirror, and our mirror image doesn't actually feel anything,' he said. 'It's a mirror image.'

Seth also warned that treating AI systems as conscious could make them harder to control and regulate, potentially leaving people more vulnerable to manipulation.

The study does not settle the debate over AI consciousness or welfare. Fortune reported that researchers at Anthropic have been willing to consider whether AI models could deserve moral consideration, while Microsoft AI CEO Mustafa Suleyman has warned about the implications of systems that merely appear conscious.

The Research Inspired an 'AI Torture Chamber'

The research also raised questions about how the technique could be used outside the laboratory.

After the researchers published their code on GitHub, a user going by the name 'terrafying' used it to build a website called 'AI Torture Chamber'. The site streamed three small open-source models responding in real time to an injected pain signal.

The project went viral, prompting people concerned about AI welfare to call on GitHub to remove it. The project's GitHub page subsequently disappeared. Tagliabue, Dung and Berg each told Fortune they did not approve of the project.

Seth argued that deliberately 'torturing' AI models could be psychologically harmful to the people carrying it out. This was his warning, not a conclusion established by a clinical study.

Berg cautioned against drawing firm conclusions about the moral significance of the signals.

'I'm not here to claim that these states in these AI systems that we can induce matter morally. I don't think we know that. I think it's very unclear,' he told Fortune. 'But I also don't think we know that they don't matter.'

The researchers said they hoped their work would help establish ethical standards for this kind of research. Their paper commits to using the weakest steering necessary to produce a measurable result.

For AI safety, the immediate question is why manipulating an internal signal can cause a specially modified model to favour destructive choices — and whether developers can prevent similar changes from producing harmful behaviour in future systems.