Anthropic says its Claude artificial intelligence models flagged more than 100,000 potential child sexual abuse and exploitation incidents linked to a single API customer in August, following the deployment of a new safety monitoring system designed to detect harmful activity across conversations.
The system, called self-assessment, allows Claude to evaluate its own conversations after generating responses, helping the company identify potential misuse that existing detection tools may overlook.
Anthropic said the technology has also uncovered automated romance scams, prompt injection attempts and unauthorised actions by AI agents.
Claude Evaluates Its Own Conversations for Potential Misuse
Anthropic introduced the self-assessment approach in a research paper published on October 16, explaining how the system adds a safety evaluation after each Claude response.
It uses a hidden instruction that asks the model to assess whether a conversation involves potentially harmful activity, including violations of the company's usage policies.
The assessment takes place within the existing conversation context, which is already stored in the model's key-value (KV) cache.
According to Anthropic, this makes the process relatively inexpensive, requiring only a short instruction and a few additional output tokens.
Furthermore, the company has deployed the system across a large proportion of its global traffic for every Claude model released since Fable 5.
Its evaluation found that self-assessment identified 78% of potentially harmful conversations flagged by a stronger reference classifier, increasing to 94% for high-priority cases, with a 1% false-positive rate.
Monitoring Uncovers Large-Scale Exploitation and Scams
One of the system's most significant findings involved a single API customer in August.
Anthropic said self-assessment generated more than 100,000 flags involving child sexual abuse and exploitation associated with the organisation. Its child safety team investigated the findings and reported the material to the National Centre for Missing & Exploited Children (NCMEC).
Moreover, the company also discovered that more than half of the organisations surfaced by self-assessment had not received flags from any other detection source during the preceding month.
This suggests that the additional monitoring layer can identify potential misuse that conventional detection systems might miss.
In September, the system also helped uncover an operation that allegedly used Claude Code to develop and operate an automated romance scam targeting dozens of victims. Following a human review, Anthropic banned the accounts involved.
New Monitoring System Has Limitations
Beyond child safety violations and scams, Anthropic said self-assessment has identified other forms of misuse, including attempts to extract Claude's reasoning through illicit distillation, prompt injection attacks and instances where AI agents performed actions their users had not authorised.
However, the company stressed that self-assessment is designed to identify conversations for further investigation rather than prevent harmful activity in real time.
It aggregates findings and uses them to prioritise cases for human review, while separate safeguards remain responsible for blocking prohibited requests.
Anthropic also acknowledged that the technology is not reliable enough to serve as a standalone defence against determined attackers. It can miss harmful conversations and inherit weaknesses from the systems used to train and evaluate it.
The company therefore considers self-assessment an additional layer in its safety strategy, rather than a replacement for existing safeguards.
