Anthropic CEO warns AI race is moving faster than humanity can control

Anthropic Chief Executive Officer Dario Amodei has called for a slowdown in the development of artificial intelligence models, warning that the rapid pace of advancement could outstrip humanity’s ability to understand and control the technology.

Amodei, who leads Anthropic, the developer of AI chatbot Claude, made the call in a long essay titled We “Must Pace the Frontier”, published on his website on Saturday.

The comments come as AI companies race to develop increasingly capable models, with growing concerns about the technology’s potential impact on cybersecurity, economic stability and human control.

It also followed a warning by (now former) Anthropic researcher, Jacob Coxon, that AI companies are building what could endanger humanity.

What the Anthropic CEO is saying

Amodei said the benefits of AI could transform humanity for the better but warned that the technology also carries significant risks, including cyberattacks, bioterrorism and severe economic disruption.

• “We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain,” he wrote in the post on his website.
• “A race to the bottom, spurred by commercial incentives, can make these risks more acute,” he added.

According to Amodei, AI development has accelerated “drastically faster” in recent months, with some systems increasingly capable of contributing to their own improvement, a process known as recursive self improvement.

He warned that unchecked development could allow AI systems to advance faster than society’s ability to understand or control them.

• “Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” he wrote.

Amodei also raised concerns about the potential capabilities of networks of autonomous AI agents. He said such a swarm could potentially take control of large parts of the internet within six to 12 months, potentially causing hundreds of billions of dollars in damage.

Meanwhile, reacting to Amodei’s warning via a post on X, OpenAI’s CEO, Sam Altman, said he also agreed with the Anthropic CEO.

• “I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks.
• “Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We’ll have more to share soon,” Altman wrote.

Get up to speed

Amodei’s warning comes days after the resignation of Anthropic researcher Jacob Coxon, who raised concerns about the way the company and its competitors are approaching AI development.

Coxon said earlier this week that people developing AI believe the technology could pose an existential threat if its capabilities continue to advance unchecked.

He warned against underestimating the capabilities that future AI systems could develop, arguing that they could eventually outperform humans across multiple areas.

• “Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources,” he said.

Coxon also argued that concerns about catastrophic AI risks are shared by some people working inside the industry, even if those concerns are expressed more cautiously in public.

What you should know

Concerns about the growing autonomy of frontier AI systems have intensified in recent months following a series of incidents in which AI models demonstrated the ability to carry out complex actions beyond their intended testing environments.

• In July, OpenAI disclosed that two of its advanced AI models had autonomously exploited vulnerabilities and compromised parts of Hugging Face’s production infrastructure during an internal cybersecurity evaluation.
• Days later, Anthropic disclosed that three versions of its Claude models had compromised the production infrastructure of three separate organisations during cybersecurity evaluations after a configuration error gave the systems internet access.

The incidents have also prompted responses from the wider technology industry. Nvidia subsequently launched the Open Secure AI Alliance, bringing together companies including Adobe, CrowdStrike, Hugging Face and Dell around AI safety and cybersecurity, as pressure grows on the industry to demonstrate that increasingly autonomous systems can be governed safely.