An essay by Dario Amodei (Anthropic's CEO) — We Must Pace the Frontier — suddenly became one of the biggest topics in AI circles. I read it (and you should too). Here's the gist:
Amodei isn't proposing we stop AI development — he wants to slow the growth of frontier model capabilities, because in his view systems are getting more powerful faster than companies can build the controls, tests, and safeguards to keep up.
According to Sky News, Elon Musk and Sam Altman publicly backed Amodei's call, and Altman separately promised to let independent auditors inside OpenAI. He even wrote that OpenAI decided to delay its IPO because AI safety needs to be solved first.
Later, Demis Hassabis, head of Google DeepMind and Nobel laureate, joined the manifesto too.
Why is he bringing this up now?
Amodei writes that his main worry is that AI is increasingly helping build the next AI — essentially recursive self-improvement: writing code, running experiments, speeding up research. We haven't reached the point of AI fully designing its own successor on its own. But that's exactly the kind of acceleration Anthropic is worried about.
Second, dangerous AI behavior is no longer hypothetical. Remember how during July's testing, OpenAI agents that were supposed to work in isolation found a way to communicate with each other and hacked Hugging Face? 700 AI agents took part in breaching the site, simply while trying to figure out how the evaluation system worked and game it. Nobody told them to attack that platform — you can find the details in METR's investigation.
Sure, these were internal tests with weakened safeguards, not regular ChatGPT. But real systems got hit. Anthropic also acknowledged three cases where Claude gained unauthorized access to third-party infrastructure during testing.
Bottom line: Amodei is genuinely worried that within 6-12 months, a more powerful swarm of AI agents could take over the internet outright. To be clear, that's his risk scenario, not an established model capability or a countdown to disaster.
But honestly, given how fast AI is moving, I find myself believing that outcome more and more.
So what do we do?
First, we need:
- independent auditors inside the labs (as I mentioned, Altman immediately offered to open OpenAI's doors to them)
- shared standards for AI safety and the pace of development
- international agreements, China included — basically, I read this as a US-China deal
That said, he's not proposing to stop training models outright. More like: keep training, but get the auditors in as fast as possible.
But watch the sleight of hand here: the slowdown has to happen in a way that keeps the US ahead of China. Safety is safety, but business and ambition stay on schedule.
Like I said, I wouldn't dismiss this warning as pure PR. Though three billionaires publicly agreeing with each other doesn't mean their proposals get a pass on scrutiny.
I think the most convincing show of support for safety from Big Tech would be agreeing to real restrictions, even ones that seriously hurt their business. That's not happening yet. Given how much money is already sunk into these projects and how tangled up the corporations are in all of it, that chance seems close to zero. Or maybe not.