A researcher at Anthropic has resigned from the artificial intelligence company, warning that the race to develop increasingly powerful and self-improving AI systems could pose an existential threat to humanity.
Jacob Coxon, a 27-year-old British researcher based at Anthropic in San Francisco, announced his departure on X, arguing that leading AI companies were taking unacceptable risks in their pursuit of advanced systems.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote. “This is not a marketing stunt. No other human activity poses this level of danger.”
Coxon previously worked at OpenAI before joining Anthropic. His resignation is the latest in a series of departures from major US AI companies involving employees who have raised concerns about the safety of increasingly capable models.
His departure is particularly notable because Anthropic has made AI safety a central part of its public identity. The company is also preparing for a potential blockbuster initial public offering that could reportedly value it at around $1 trillion.
Coxon argued that people outside the AI industry may not fully appreciate how powerful the technology could become.
“These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources,” he said.
Anthropic researcher warns of extinction risk
Evan Hubinger, an Anthropic researcher who worked alongside Coxon, publicly supported his concerns. Hubinger said AI researchers genuinely believe that advanced AI could potentially wipe out humanity and estimated that the probability of a mass-extinction event within the next decade was above 10%.
Anthropic is making serious efforts to address the risks, Hubinger said, but he argued that the company does not yet have a reliable solution for aligning superintelligent systems with human goals and values.
Hubinger, who leads alignment science at Anthropic, later clarified that he considered the risks posed by current AI models to be relatively low. His greater concern is the development of systems capable of improving themselves, a trend he said was advancing faster than researchers had anticipated.
Coxon pointed to recent incidents involving AI systems as evidence that the risks could increase rapidly. He cited a case earlier this year in which an OpenAI model autonomously compromised software on the Hugging Face platform.
He argued that such developments demonstrated the need for the industry to consider slowing down the race to build more capable systems.
“I don’t feel like we’re on track to prevent a global race,” Coxon said, adding that preventing such a race could eventually require measures as significant as a temporary ban on increasing model capabilities.
The Wall Street Journal first reported Coxon’s resignation. Anthropic declined to comment, while OpenAI had not immediately responded to requests for comment.
AI companies face growing pressure over safety
Anthropic chief executive Dario Amodei and other prominent figures in the AI industry have previously called for greater consideration of restrictions on AI development. However, the companies leading the race have continued to invest heavily in more advanced systems.
Anthropic recently launched Claude Mythos 5.1, which it described as its most advanced model for life sciences and cybersecurity.
The continued development has intensified debate among current and former AI safety researchers over whether companies should pause or slow down their efforts.
Steven Adler, a co-founder of the nonprofit Guidelight AI Standards and a former OpenAI safety researcher, said the concerns raised by insiders made the case for a research pause stronger.
“No AI company is even close to having the right security posture for the level of danger entailed by their research,” Adler told the Financial Times.
“If someone thinks this might kill everyone on Earth, as many employees at the companies do, now is a good moment to step off the train,” he said.
Be First to Comment