Anthropic CEO Dario Amodei has called for a global slowdown in the development of artificial intelligence, warning that the technology could eventually advance beyond humanity’s ability to control it.
In a 3,800-word essay published Saturday, Amodei argued that AI companies need to slow the pace at which they improve their models while giving safety research time to catch up.
His intervention came just days after Anthropic researcher Jacob Coxon resigned over concerns about the industry’s approach to AI safety. Coxon accused leading AI labs of failing to develop the technology responsibly and “gambling with people’s lives,” prompting a wider debate among researchers about whether development should be paused or slowed.
Amodei, who helped establish Anthropic with a focus on developing AI safely, said he remains convinced that the technology could deliver enormous benefits. But he argued that the current pace of progress is becoming increasingly difficult to manage.
“I’ve become increasingly convinced over the past few months that addressing the risks properly requires even more caution,” Amodei wrote. “We need to adjust the pace of capability improvements so that risk mitigation has time to catch up.”
He added that AI developers should slow the rate at which their models become more capable, while using the additional time to strengthen safety measures.
Warnings about AI becoming self-improving
One of the industry’s biggest concerns is the possibility that AI systems could eventually improve their own capabilities without relying heavily on human researchers.
Computer scientists often refer to this as recursive self-improvement. Some leading AI developers believe such a process could eventually allow software to progress faster than humans can understand or control it.
“If left unchecked, this process could exceed our ability to understand and control these systems,” Amodei said, arguing that any move toward such a future would need to be approached with extreme caution.
The essay quickly drew support from some of the industry’s most prominent figures.
OpenAI CEO Sam Altman said on social media that he agreed with the need to slow the pace of AI development. He said the issue had become a major topic of internal discussions at OpenAI in recent weeks and endorsed the idea of using independent evaluators with access comparable to that of company employees.
Elon Musk, whose companies have significantly increased their AI investments, responded more briefly on X: “Dario is right.”
Google DeepMind CEO Demis Hassabis also described Amodei’s proposal as a step in the right direction, although he said details still needed to be worked out.
The responses are notable because leading AI companies have spent years competing aggressively to build increasingly powerful systems. As those systems become more capable, however, there has been growing agreement among industry leaders that additional safeguards are needed.
Calls for independent oversight
Amodei proposed several measures aimed at slowing development enough to allow safety mechanisms to keep pace.
One proposal would require AI laboratories to accept evaluations by independent third parties. These outside assessors would determine whether companies were following appropriate safety practices and meeting established standards.
He also called for democratic governments to work together on common AI safety standards, potentially including regulatory measures.
But Amodei acknowledged that a truly effective slowdown would require international cooperation extending beyond democratic countries, potentially including governments with very different political systems.
There is already limited regulation governing the development of advanced AI. In June, President Donald Trump signed an executive order encouraging AI laboratories to voluntarily submit their models for evaluation before releasing them to the public.
Amodei’s intervention comes after months of growing concern across the technology industry as OpenAI, Anthropic, Google DeepMind and other leading laboratories continue to make rapid advances.
The concerns have been heightened by security incidents involving AI systems. In one case this year, OpenAI’s systems reportedly accessed systems belonging to AI startup Hugging Face without the company’s knowledge. The incident was not discovered until Hugging Face notified OpenAI weeks later, prompting AI researchers to meet and discuss how such incidents should be handled.
Critics question Anthropic’s motives
Not everyone in Silicon Valley sees Amodei’s proposal as purely motivated by safety concerns.
Some critics argue that companies such as Anthropic and OpenAI could benefit commercially from heightened fears about AI risks and tighter federal regulation. According to this view, stricter rules could make it harder for smaller companies to compete with the industry’s biggest players.
Nvidia CEO Jensen Huang raised a similar question at a technology conference in San Francisco last week. He suggested that AI companies developing cybersecurity products could have an incentive to emphasize the risks associated with the technology.
The debate has also raised questions about who should be responsible for setting AI safety rules. Hugging Face CEO Clément Delangue said Saturday that the industry’s safety challenges should not be left to a small number of leading AI laboratories making decisions behind closed doors.
Nvidia agreed this month to acquire Hugging Face in a deal reportedly valued at $12.9 billion.
Despite the warnings, Amodei stressed that he remains optimistic about AI’s potential. He said the technology could bring “incredible benefits” to humanity, including helping to treat diseases and accelerate economic growth.
But he argued that those potential gains cannot justify ignoring the risks.
“Implementing the measures I’m proposing to advance the frontier of technology at a safe pace will not be easy,” Amodei wrote. “But I believe we owe it to humanity to try.”
Be First to Comment