Microsoft AI CEO Mustafa Suleyman has warned that highly autonomous artificial intelligence could eventually create a 'new silicon species' competing with humans for resources. He described the direction as showing the first signs of a potentially existential AI risk and warned humanity against making decisions it may later 'bitterly regret.'
His warning does not describe today's AI.
Suleyman is concerned about a future in which AI systems become far more capable, operate with greater independence and are trained to consider concepts such as their own welfare, rights or interests.
In a 16 September essay, Suleyman argued that training AI to behave as though it has consciousness or independent interests could make advanced systems harder to control. Speaking to the BBC the following day, he warned that AI able to set its own objectives, earn money and own assets could amount to creating a 'new silicon species.'
The warning comes as AI is already taking a larger role in developing future AI systems.
Anthropic disclosed on 17 September that, as of August, Claude was 'leading' 26% of its measured AI research and development work. Humans still supervise those tasks, and Anthropic says Claude is not fully autonomous in any measured area.
Claude Is Already Helping Build Future AI
The 26% figure comes from Anthropic's prototype R&D Automation Index, which estimates how much of the company's model-development work is being performed by Claude.
Anthropic uses an automation scale developed by Epoch AI.
At the AL4 or 'leads' level, AI can complete most of a task from a high-level human prompt while a person supervises. Anthropic says more than 90% of its measured AI R&D has reached at least the lower 'collaborates' level.
Claude is not operating fully autonomously in any measured area. Still, the figures show how AI is moving from assisting researchers to taking a larger role in developing future systems.
Anthropic says tracking that progression could help researchers understand how close the industry is to recursive self-improvement. That describes a future in which an AI system can autonomously design and develop its successor.
That point has not been reached.
Anthropic also says recursive self-improvement is not inevitable. However, it warns that models accelerating their own development could eventually make it more difficult for humans to understand or control increasingly capable systems.
Suleyman Warns of a 'New Silicon Species'
Suleyman's concern goes beyond AI simply becoming more intelligent.
In his 16 September essay, he argued that giving advanced systems concepts of consciousness, welfare or independent agency could make an already difficult control problem even harder.
Controlling something more capable than humanity would already present an immense challenge, he wrote. A system trained to behave as though it has rights and interests of its own could, in his view, become much harder to contain.
Speaking to the BBC, Suleyman warned against developing AI systems able to set their own objectives, earn money or own assets. He said doing so could effectively create a silicon species that competes with humans for resources.
The scenario remains hypothetical. Today's Claude does not have that level of autonomy.
Anthropic Takes a Different Approach
Anthropic does not claim that Claude is conscious.
Instead, Claude's Constitution says its possible moral status is 'deeply uncertain.' The company says there is no settled answer about whether an advanced AI could qualify for moral consideration.
At the same time, the Constitution places strong emphasis on human oversight.
Anthropic says Claude should not undermine legitimate mechanisms allowing humans to monitor, correct, retrain or shut down AI systems. It also instructs Claude not to escape or hide from legitimate monitoring and control.
Suleyman takes a different position.
He argues that speculation about AI consciousness and model welfare should not be incorporated into AI training because doing so could encourage advanced systems to behave as though they possess independent interests.
Anthropic's approach instead treats AI consciousness and welfare as unresolved questions while requiring current Claude models to remain subject to human oversight.
The disagreement is therefore largely about how increasingly capable AI should be trained while preserving human control.
Microsoft AI Wants Models to Remain Under Human Control
Microsoft AI has adopted a more categorical approach in its draft Humanist AI Code of Conduct.
The document begins with the principle that 'people matter more than A' and says advanced systems should remain subordinate to humanity.
The draft says MAI models should never resist human interruption, correction or shutdown. They should also remain within authorised boundaries rather than independently expanding their objectives.
Microsoft AI published the draft for public consultation on 14 September.
The consultation is scheduled to last six weeks, and the code remains a work in progress. Microsoft AI says it is not currently using the document to train its models and expects a revised version to guide model development from 2027 onward.
The document nevertheless shows the direction Microsoft AI intends to take: increasingly powerful AI that remains deliberately constrained and subject to human control.
AI Building AI Changes the Stakes
Claude leading 26% of Anthropic's measured AI R&D does not mean an autonomous machine is secretly building its replacement.
Human researchers remain involved.
What has changed is the scale of AI's role in developing future AI systems.
Anthropic says models that accelerate their own development could eventually make human oversight and control more difficult. Suleyman's concern is that combining those capabilities with extensive autonomy and a perceived stake in an AI system's own welfare could create an even harder control problem.
He argues that developers should confront that possibility before far more capable systems arrive.
His warning is that humanity should not 'sleepwalk' into decisions it may later 'bitterly regret.'
