Listen to the article
King Charles has warned about the dangers posed by artificial intelligence as technology executives and researchers call for greater safeguards around increasingly advanced AI systems.
The King said: “Those who have created these technologies are now increasingly warning that AI risks developing darker capacities – perhaps even to take life.”
He said there was an “urgency” in “adequately considering the existential dangers of such technologies falling into the wrong hands, and being used in potentially catastrophic ways”.
He added: “The task before you is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world.”
The comments were made at a summit convened by the King at Dumfries House in Scotland, bringing together senior figures from government and the technology industry, including representatives from Nvidia, OpenAI and Anthropic.
Mustafa Suleyman, chief executive of Microsoft AI, has separately warned against developing AI systems capable of setting their own objectives, earning money and owning assets.
Suleyman said doing so would amount to “essentially seeding a new silicon species”.
He warned such systems could “no doubt compete with us for resources, no matter how much it cares about humanity and loves us”.
Suleyman said it was “right” for people to be concerned about AI, but said there were “very practical things that we can do” to control it.
Evan Hubinger, an AI alignment researcher at Anthropic, said this month that he believed there was a greater than 10 per cent chance AI “could kill all humans” within the next decade.
Hubinger wrote: “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.
“I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”
Hubinger said the risk from existing AI systems was “low”. His warning concerned the development of more advanced systems and the possibility of AI helping to create subsequent generations of the technology.
Anthropic researcher Jacob Coxon has also said he and colleagues were “genuinely frightened” about the speed of AI development and its potential consequences.
Anthropic chief executive Dario Amodei warned that AI had advanced “drastically faster” than expected, including in its “ability to build the next generation of AI”.
Amodei wrote: “We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.”
OpenAI chief executive Sam Altman has also called for the development of advanced AI to be paced.
Altman said at a Salesforce conference in San Francisco: “It doesn’t take as much imagination as it used to for [us] to imagine how this could go wrong. I think the world is right to be afraid of this.”
He added: “The world should trust that we are going to do the right thing because it’s the right thing and we feel the magnitude of this.”
Altman has said that pacing development should not mean stopping technological progress.
He said: “Progress has been rapid and will continue to be.
“But it should be slower than it otherwise could be – interventions like safety cases and monitoring have significant costs.”
Suleyman has also criticised attempts to give AI systems human-like characteristics and treat them as though they possess their own desires, values or identities.
In an essay published this week, Suleyman wrote: “AIs are not conscious.
“They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations.
“They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans.”
Suleyman called for greater transparency around how AI systems are trained and evaluated, independent scrutiny of their behaviour and stronger mechanisms to monitor and control them.
He wrote: “We must not sleepwalk our way into a decision we later come to bitterly regret.”
Microsoft established a superintelligence team in October 2025 and has published a Humanist AI Code of Conduct setting out its approach to developing advanced AI.
The company describes “humanist superintelligence” as “very advanced AI that always works for people, stays within limits, and remains under human control”.
Suleyman said: “I think that the good news here is that everybody who is human is going to have a very strong interest in making sure that the systems that we all create and… are used around the world in every nation are safe and controllable and subordinate to humanity. Everybody must be aligned.”
Nvidia chief executive Jensen Huang, who attended the King’s summit, said AI safety was “paramount” and that companies should “keep engineering” if a product was not sufficiently safe.
Huang said: “Let us lead with responsible optimism, build AI safely and securely, open it to more people, expand our ambition and solve the problems that matter to the world.”
The warnings have prompted calls for greater regulation of advanced AI, although there is disagreement over how governments should regulate the technology and whether AI companies can adequately police themselves.
