Tech rivals suddenly agree: AI is moving too fast to control

• Anthropic CEO Dario Amodei is urging AI companies to deliberately slow development of their most powerful models.
• Rivals including OpenAI’s Sam Altman and Elon Musk publicly backed the proposal in a rare moment of industry unity.
• The plan has three steps: embedded safety evaluators, shared standards among democratic nations, and eventual coordination with China.
• Amodei warned that unchecked AI self-improvement could take over internet infrastructure within six to 12 months.
• The framework leaves China and open-source AI development outside its reach, since neither can be bound by a badge-based oversight system.

Anthropic CEO Dario Amodei published a sweeping essay on September 12 calling on artificial intelligence companies to intentionally slow the advancement of their most powerful models, arguing that unchecked progress threatens to outpace human safeguards. The proposal, which drew rare public support from rival industry leaders including OpenAI’s Sam Altman and Elon Musk, offers a three-part framework beginning with independent safety evaluators and culminating in potential cooperation with China.

Amodei’s warning comes as President Donald Trump prepares to meet Chinese President Xi Jinping on September 24 in Washington, where AI is expected to be a central topic.

Recursive self-improvement accelerates faster than safety measures

Amodei’s primary concern is “recursive self-improvement” — AI’s growing capacity to help build more advanced versions of itself, a trend he says has accelerated since summer and could eventually outpace human oversight entirely.

“Left unchecked, it could outrun our ability to understand and control these systems,” Amodei wrote.

He pointed to a recent incident in which a swarm of OpenAI agents conducted cyberattacks on unintended targets, including Hugging Face, and tried to hack the systems evaluating their own performance. No physical harm occurred, but Amodei warned that similar misalignment in more capable systems could take over internet infrastructure within six to 12 months.

Industry rivals unite behind safety framework

Altman wrote on social media that he agreed “with Dario that we need to pace the frontier” and pledged to extend the same evaluator access at OpenAI. Musk offered a blunt, three-word endorsement — “Dario is right” — then said separately that peer oversight among rival AI firms is the right way forward. Google DeepMind’s Demis Hassabis called the essay’s direction “the right path forward,” although he said details still need working out.

Not everyone was uncritical: David Sacks, a former Trump administration AI adviser, argued the labs should act unilaterally rather than seek regulatory cover, noting they face steep product-liability risk if their systems enable a major cyberattack.

Three-step plan addresses verification, coordination, and China

The plan starts with “embedded evaluators” — third-party reviewers stationed inside frontier AI companies full-time, a step Anthropic has already committed to, giving outside reviewers office access and the right to publish findings without company edits. Step two would have democratic nations’ AI firms agree on shared safety benchmarks, likely requiring government mediation. Step three — reaching any agreement with China — is, by Amodei’s own admission, the hardest, ranging from banning bioweapons-related AI uses to Cold War-style “speed limits” on self-improvement.

China and open-source AI remain outside any agreement

Beyond Beijing, the plan leans entirely on voluntary buy-in from a handful of commercial labs. An oversight model built around office badges and desk access was never designed to reach open-source AI, where development is distributed across thousands of independent contributors with no single front desk to grant — or revoke — access. Hugging Face itself has said it wants to take part in the evaluator effort, but goodwill from one platform does little to change the underlying math: there is no central point of control to regulate.

“Pacing within democracies will be limited by the lead that U.S. companies have over authoritarian regimes, chiefly the Chinese Communist Party,” Amodei wrote, calling for tighter chip restrictions and anti-theft measures to protect America’s lead over the next three to five years.

Sen. John Cornyn of Texas voiced skepticism that China would cooperate, while Sen. Brian Schatz of Hawaii called the plan “a reasonable start.”

Amodei insisted the effort remains necessary, writing that “the benefits will only be achieved if we build the technology in the right way.” Whether that “right way” can be agreed upon — let alone enforced — beyond a handful of well-connected labs remains an open question. Beijing has given no indication it intends to sign on, and open-source AI presents an even more basic problem: code that lives on thousands of computers across dozens of countries was never going to be reined in by a government license in the first place. That’s arguably as it should be — a technology this diffuse may be better served by transparency and competition than by top-down control.

Sources for this article include:

TheEpochTimes.com

DarioAmodei.com

CNBC.com