Synopsis: A resignation, two missiles, and a page of small print — and what they reveal about an AI industry that has learned to confess the dangers its very existence poses without pausing for breath. AI leaders openly warn that the technology they are building could end humanity. Then they go back to work and accelerate it. Here is why none of them can afford to stop.
Jacob Coxon sat on a bench in Alamo Square and posted, to a stranger’s platform, that he had resigned from Anthropic that morning. Three years split between Anthropic and OpenAI, most of it spent on the invisible discipline of pre-training — that part of the research which decides what a model will have read before it is allowed to decide anything. Coxon wrote that both firms were sprinting toward systems they could not fully steer or control, and that he no longer wanted a hand in it.
The post reached tens of millions of people within a day. In it, Coxon put his estimate in plain figures: better than a one-in-ten chance that the technology he had spent three years building would, within the decade, bring about the extinction of the species that built it. Two days later, Anthropic’s own alignment lead, Evan Hubinger, said publicly that he shared the number.
Then he went back to his desk. So did everyone else.
One detail in Coxon’s resignation reveals more than the headline. He quit two months before his equity in the company matured. That detail compresses the whole story into a single refusal: a warning that costs something to deliver reads differently from one that costs nothing. And yet it changed nothing measurable about the pace of R&D in either company. Training continued. Both firms remain on course for the largest public listings in the industry’s history. This is what conscience looks like inside an industry it can’t slow — sincere, extremely expensive to the person holding it, yet trifling to the business itself.
A fairly dramatic shift has overtaken the category of whistleblowing. It used to imply an intention to stop, or at least a willingness to be stopped — a line a person crossed once and could not cross back over. What Coxon and Hubinger produced instead sits closer to a public accounting of risk that both men continued to help generate from the inside: a confession offered mid-act, from people still feeding the machine. The arithmetic, once believed, argues for exactly one conclusion; neither man stated it out loud. The confession asked only to be believed. And belief, it turns out, is cheap to extend and cheaper still to act on.
Tobacco executives who knew what inhaling nicotine did. Oil companies that modelled the coming heat decades before they funded any misgivings. The pattern is not new. What’s new is the speed with which confession and continuation now arrive in the same week, sometimes in the same sentence, without embarrassment on either side.
Seven thousand miles from that bench, on the opening day of a very different acceleration, a purported failure killed more than a hundred and twenty children in a schoolyard in southern Iran.
Two missiles struck an elementary school in the town of Minab — a building that commercial satellites had shown, for years, painted in the bright colours of a primary school, ringed by a wall someone had built specifically to separate it from the military compound next door. A later Pentagon review found that officers approving the strike had leaned on an automated targeting platform, built by Palantir, that compressed what should have been days of cross-checking into minutes.
No one, to my knowledge, has claimed the system acted alone. That’s exactly what gets missed by everyone still debating whether some future AI might someday decide to kill us. This one already helped — by making the humans in the loop faster, and less inclined to ask what they had missed.
And the case didn’t stay a single failure. On the first of September, missiles hit a wedding party in Kuhestak, twenty miles down the coast from Minab. Iranian officials and the UN attributed the strike to the same actor. The file remains open. Nobody built a second dashboard for that one. Whatever had shortened the distance between suspicion and strike in February was still running in September, unaltered, because the review that had found it had not yet finished being written.
There’s another, far more dispassionate kind of evidence than either of these stories displays. It comes from the actuarial profession - people paid to price risk rather than argue about it.
For years, the exposure sat inside ordinary policies without a name — general liability and errors-and-omissions forms written before the technology existed, stretched by default to cover it. Insurers called this silent cover, and it worked in the policyholder’s favour for exactly as long as nobody looked too closely.
Since the start of this year, they have started looking. The Insurance Services Office, the body whose language most American carriers build on, issued endorsements in January that exclude bodily injury, property damage and reputational harm arising from generative AI. The exclusions are written broadly enough that a claim need not be centrally about the technology to trigger them — only connected to it in some way. Chubb, Berkshire Hathaway and Travelers have each won regulatory approval to attach the exclusion to their standard commercial policies. State regulators have approved the overwhelming majority of such requests. Meanwhile, lawsuits citing generative AI grew by close to a thousand per cent between 2021 and 2025. So the actuaries drew the only conclusion their discipline permits: this is no longer a risk they can underwrite.
None of this is announced as a verdict on whether the researchers are right or wrong about the odds. It functions as a judgement anyway — and a more forthright one than anything said in public. A business willing to underwrite almost any risk for a price is refusing to underwrite this one at any price it can charge and still call itself a business. Which is another way of saying that the people whose entire trade is pricing disaster have quietly concluded some of it may not be priceable.
What replaces coverage is not caution but redirection: governance checklists, indemnity clauses buried in vendor contracts, a widening expectation that whoever used the tool, not whoever built it, will be standing there when the claim is filed. The same month insurers began pulling back, both companies were still being valued as though no such tail existed — priced for the upside, uninsured for the rest. Capital has read the same arithmetic Coxon and Hubinger described, reached a similar private conclusion. They have not asked anyone to slow down. They have simply made sure that when the failure comes, someone else pays for it.
Coxon’s fear, the deaths in Minab and Kuhestak, and the quiet retreat of capital cannot be flattened into the same order of damage. The first is speculative but has potentially catastrophic implications. The second is documented - already counted and buried. The third is a number on a renewal form nobody outside the industry reads. But they share an anatomy. In each, judgement was compressed by a structure that rewarded speed over verification — and everyone inside that structure, from the army officer with seconds to approve a missile strike, to the researcher with two months until his shares matured, could describe the compression clearly enough. Nothing and nobody could slow it down alone.
You don’t have to believe any of these people are dishonest, crazy, or even unusually compromised. After all, a firm that slows down while its rivals do not doesn’t make the technology any safer — it simply hands the next model, the next contract, the next round of funding to whoever kept going. Restraint offered unilaterally does not subtract risk from the field. It subtracts the restrained party from it, and leaves the risk exactly where it was, now in less cautious hands.
That’s not a flaw in any one company’s character. It’s the arithmetic every company in the race can do, correctly, in the time it takes to read this sentence.
There’s a much smaller case that shows what changing the structure actually entails, and it has nothing to do with intelligence. Danish building firms were asked, a few years back, to cut their greenhouse gas emissions well below the legal floor. They tried it one company at a time. Every firm that held itself to the tighter standard lost the next tender to a rival who hadn’t even bothered. Restraint, attempted alone, produced exactly its opposite. What changed the outcome was not a change of heart among the reluctant. It was six hundred competitors — later closer to nine hundred — asking their own parliament to bind every one of them at once, so that virtue would stop costing a contract. Denmark tightened the cap in 2024. No single firm chose to be virtuous. The industry changed the terms under which virtue could survive being chosen.
It would be a comfort to end there, with a model ready to export to an industry racing toward something considerably larger than a construction tender. But concrete pours the same in Copenhagen as anywhere, and the comparison has a limit worth stating. A parliament with jurisdiction over its own builders is not a stand-in for a handful of firms answering to shareholders in several countries, competing against states that share none of their incentives. What they’re building, if it fails, doesn’t just miss a deadline. It fails once, permanently, for everyone on this planet. The gap between the two cases matters more than the resemblance. Whether an equivalent structure can be built for a technology with no single regulator and rivals who owe each other nothing is a question I am neither equipped or prepared to answer.
A man forfeited a pile of money to say, once, cleanly, that he no longer wanted a hand in what he had helped build — and his work carried on regardless that same afternoon. A school stood painted and photographed for years before anyone matched the pictures to the target file, and the same failure was already repeating twenty miles down the coast before the report on the first catastrophe was finished. Underwriters, who owe nobody an opinion, have quietly reached the same number the researchers reached, and made certain it will not be their money that proves them right.
The pattern that connects a resignation, a pair of missiles, and a page of small print is not that a machine might one day decide to decimate humanity. It’s that the humans inside these systems can now describe, accurately and in public, exactly what they are doing and why it’s so dangerous — and the description has stopped functioning as a brake. Confession has been absorbed into operation.
The danger was never an artificial mind deciding to act without our permission in spite of the fact that it’s been portrayed like that in numerous Hollywood movies. The greater threat comes from a species that has learned to narrate its own recklessness in full and to continue with business-as-usual anyway — fluently, on schedule, blindly, and without any of the shame that used to make honesty cost something.