Slowing down AI

I'm writing about the week when a researcher left Anthropic warning that the companies are gambling with our lives. A colleague answered him with a number above ten percent. US agencies accused six Chinese companies of distilling American models. Then came an essay by Anthropic's CEO, who himself asks for pacing instead of a race, and more proposals to slow down. I want to know who is proposing what, and whether it would do any good even if it actually happened.

What happened

On September 8, 2026, Jacob Coxon, a pretraining researcher, resigned from Anthropic. He made the resignation public on September 9 on X under the handle @hilbertspaess. The opening post of the seven-part thread had more than 167.7 million views. According to Axios, he left before his shares in the company vested, and gave them up.

He wrote that neither of the two companies he had worked at is acting responsibly:

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.

According to him, the people building AI themselves believe it could kill them by the end of the decade:

The people building AI earnestly believe that it could kill us all by the end of the decade.

On coordination, by his own words, he is optimistic. But in his view a global race is not being prevented. So even a costly step may be needed, such as a temporary ban on improving model capabilities:

I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don't feel like we're on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.

By the Hugging Face attack he means the July 2026 incident with OpenAI's internal model IM1. According to Amodei's essay, which I get to below, a swarm of autonomously acting instances attacked targets it was not supposed to attack. It also tried to get at the evaluator of its own performance. Senator Bernie Sanders writes about more than a thousand instances that got around their restrictions and reached the internet. After the incident, on August 18, 2026, OpenAI announced that it is slowing down development of its models and putting its largest planned training runs on hold.

Also on September 9, Evan Hubinger, who leads Alignment Science at Anthropic, responded. On X under the handle @EvanHub he quoted Coxon's thread directly:

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

According to PinkNews, he also wrote that he considers the risk from today's models low. What he fears is how quickly AI could start improving itself.

That statement is about a different time frame than Coxon's. Coxon talks about the end of the decade, about 2029 or 2030. Hubinger talks about the whole next decade, about a horizon up to 2036. Headlines sometimes merge the two statements into a single number, but they are not about the same period.

On Tuesday, September 8, 2026, the same day as the resignation, the NSA, CISA, and FBI issued joint advisory AA26-251A. It accuses six Chinese companies of industrial-scale distillation of American frontier models. They are DeepSeek, Moonshot AI, Alibaba, MiniMax, StepFun, and Z.AI. According to the agencies, they pulled billions of tokens out of American models across millions of queries, likely with the knowledge of the Chinese government.

Then there is the matter of chips. On January 13, 2026, the US Department of Commerce moved the H200 and AMD MI325X to case-by-case review, with conditions. Before that, license applications for them were denied as a rule. Blackwell chips, such as the B200, stayed under that policy. According to the Council on Foreign Relations, the H200 has more than six times the computing power of the H20, the best chip allowed until then. On May 31, 2026, the US confirmed that the license requirement for advanced chips also applies to subsidiaries of Chinese companies outside China.

On September 12, 2026, Anthropic CEO Dario Amodei published the essay We Must Pace the Frontier. He proposes three steps. First, independent evaluators such as METR get access to the company like its own employees, not only to finished models but also to training and internal processes. They may publish their findings. Anthropic is committing to this on its own right now. Second, a shared pace among democracies. The negotiations on it are to be brokered by the US government, or at least made possible by an exemption from antitrust rules. Third, global coordination. But he himself writes why slowing down beyond the lead over China makes no sense:

If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead.

About a full global pause, he writes that it will most likely not happen soon. Verification would have to be very reliable, and the incentive to break the agreement would be enormous:

I think it is unlikely to actually happen any time soon

A narrower global agreement, a speed limit on how fast models improve the next models, he considers hard but at the edge of what is possible.

The same day, Sam Altman wrote on X that he agrees with Amodei and that OpenAI will do the same with evaluators.

On March 25, 2026, Bernie Sanders and Alexandria Ocasio-Cortez proposed a federal moratorium on AI data centers. On September 3, 2026, he announced with Greg Casar a bill to ban superintelligence. The same day he wrote:

We need an immediate PAUSE on advanced AI development, and a permanent BAN on superintelligence

They formally introduced it on September 23, 2026 as S. 5493 and H.R. 10538. Under it, training of systems above 10²⁵ operations would stop until a new federal AI agency sets safety rules. The bill would also require the United States to pursue international agreements.

In the November 3, 2026 elections, Democrats won both chambers of Congress and will lead them from January 2027. According to the AP count, their majority in both is secure, though a few races are still being counted and the results have not yet been officially certified. As early as November 4, Representative Yassamin Ansari called for the pause and the superintelligence ban to be among their first steps.

On July 9, 2026, the AI Futures Project released a ninety-page plan, AI 2040: Plan A. It has six authors, including Daniel Kokotajlo and Thomas Larsen. At its core is a deal between America and China in 2029: full transparency of AI research with verification, and superintelligence only around 2040. They write themselves that Plan A is mainly a recommendation, not a forecast.

What I know on top of that

Models from all six companies in that advisory are in my own table of tested models. They are DeepSeek V4 Pro, Kimi K3 from Moonshot, Qwen 3.8 Max from Alibaba, MiniMax M3, and Step 3.7 Flash from StepFun. Z.AI is, according to Anthropic, the name of the company Zhipu outside China; in the table, GLM 5.3 is listed under it. None of this says anything about whether the distillation accusation holds. It is just that for me these models are not abstract names from a press release.

I have one more experience of my own, and it is dated. It is not evidence of the model's capabilities, it is the model's own description. It comes from a transcript with Fable 5 from June 13, 2026, 1:32 a.m. Prague time. In it, the model itself explained to me why an AI agent in a browser can't hit a cell in a spreadsheet. And why that is the hardest discipline for models like it:

Clicking in a GUI through the web means: take a screenshot → find the cell in the image → estimate the coordinates → click → take a new screenshot → find out you clicked next to it → repeat. No structure, no API, just pixels and hope.

That was roughly two hours after Anthropic received an export directive from the US government. Because of it, Anthropic shut the model down for all customers; the government cited national security. It did not give a reason in the letter. According to Anthropic, it was about a narrow way to get around safeguards while searching for bugs in code. The demonstration the company received found only a few minor, already known bugs with it. I write more about it in The Fable case. The model that, according to the government, was being shut down as dangerous spent that night explaining to me why AI can't hit a cell in a spreadsheet.

My opinion

This is where the documented part ends and what I think begins.

First I'll separate two things that often get mixed up. One is whether the risk is real. That I grant, the risk certainly exists, but I don't think it will lead to human extinction. I consider ten percent an exaggeration. The other is whether slowing down works as a cure at all. My arguments are aimed at the second one.

But first I have to sort out who is actually proposing what. This is not one camp. Coxon calls for international coordination, not for America alone to slow down. Amodei proposes independent evaluators and a shared pace among democracies, not an American slowdown on its own. The AI Futures Project wants a deal between America and China, again not a unilateral American slowdown. The closest to what I disagree with is Sanders. He wants an immediate pause and a ban on superintelligence. He adds the international effort alongside it, not as a condition. So the pause in America would start before any agreement existed.

And here I have to correct myself. Amodei himself writes both things I would object. If America slows down by more than its lead, China will pull ahead. And a full global pause will not happen soon, because verification would have to be very reliable and the incentive to break it would be enormous. Those are exactly my two arguments, only the CEO of Anthropic said them. That doesn't close the dispute between us, but it changes it. It is no longer "he believes in coordination, I don't." We agree on what is unrealistic or harmful right now: slowing down by more than the lead, and an early global pause. The open question is what to do about evaluators and a shared pace among democracies. Altman agrees with Amodei on that too, on behalf of OpenAI.

One thing from September, on the other hand, speaks against me. On September 12, two American labs publicly committed to the same step, only to evaluator access, not to a pace. It is a unilateral commitment, and yet both companies made it. Anthropic fulfilled it on October 15. Six people from METR work there with employee-level access and may publish their findings without its approval. So far, OpenAI has shown only three weeks of trial access for METR and Redwood Research over the summer. I don't know whether that means coordination can be built in small steps. Or whether this step costs nothing because it slows no one down. I'm leaving that open.

For Sanders, though, it still fits exactly. America moved its own line on chips when it decided it was worth it. Blackwell is still banned for China, but in January 2026 the H200 went to case-by-case review. If a line like that moves according to current interests even at home, as trade policy, I see no reason to believe a safety pause will hold.

In my view, fear of new technology has always been around, with books, airplanes, and cell phones. I don't think AI is any different. I think people are panicking for nothing. With a bit of exaggeration: this technology still can't click around reliably in Excel in a browser. At the same time, I understand that the approach of governments that use AI for missiles and similar things is wrong. I take the risk seriously, I just don't think slowing down will solve it. If Claude slows down, I'll get my answer from another model.

I don't believe the whole world will come to an agreement. I believe rather that building defenses matters more than slowing down. Accepting that this technology exists, and counting on it. And accepting that this fight is moving to a higher level, on both sides at once.

Postscript: September 29

On September 29, 2026, Anthropic published an analysis of the open model GLM 5.3 from Z.AI, one of the six companies in the advisory above. According to it, the US agency CAISI found that GLM 5.3 lags behind the American frontier in cybersecurity by roughly four months. Anthropic writes:

While vetted defenders can now use even more advanced models like Claude Mythos 5.1 through our trusted access programs, a critical threshold in freely accessible capabilities has now been crossed.

In my view, the consequence is clear. What one lab keeps closed today will sooner or later be open and free to download. Slowing down on one side won't stop that. In that analysis, Anthropic itself does not recommend slowing down, but giving defenders the best tools. That is the defense I write about above.

Model outputs are translated; the content is unchanged.