Every bull market has a moment when the engineers quietly stop celebrating. I remember mine: November 2022, standing in a Frankfurt coworking space, watching FTX's liquidation cascade across a dozen screens. The loudest voices were calling it a blip. The people who had actually read the smart contracts were already packing their bags.
Last month, that moment arrived for artificial intelligence.
A group of current and former employees from OpenAI and Anthropic — the two most valuable AI companies on the planet — signed a public letter asking the U.S. government to build an oversight mechanism for “frontier AI developments.” That alone would be news. But read the phrasing closely. They want oversight because AI research automation is advancing to the point where the technology will soon move “beyond human understanding or control.”
I’ve read a lot of governance documents in my years auditing decentralized protocols. This one read less like a petition and more like a known-vulnerability disclosure — filed before the exploit.
These aren’t academics. They aren’t regulators. They’re the people whose stock options vest on the exact deployment speed they are now trying to slow. When the builders start begging for a brake, the market should listen.
To understand why this letter matters, you need to understand how rare it is. OpenAI and Anthropic are existential competitors. OpenAI’s ChatGPT defined the generative AI boom; Anthropic built its entire brand on safety-first alignment. Their engineers do not coordinate public communications. They certainly do not sign joint letters to Congress. The last time anything this cross-company happened was the 2023 wave of existential-risk open letters — signed by academics like Yoshua Bengio and outsiders like Eliezer Yudkowsky, not by the people training the actual models.
This letter is different. The signatories are internal. They have seen the training runs, the red-team reports, and the board decks. What they are saying is that the voluntary-commitment era is over.
The core request is deceptively simple. Employees want “developer obligations” that are binding rather than voluntary, whistleblower protections for people who raise concerns about AI risks, and an international cooperation mechanism to govern frontier AI development. The key phrase is “AI research automation” — the concern that AI systems will soon be designing better AI systems, creating an exponential feedback loop no safety team can pace.
I have spent the last decade translating exactly this kind of dense, technical panic for non-technical audiences. In 2017, I built ChainLit to simplify crypto whitepapers during the ICO madness, and I distributed five hundred copies to student clubs, watching the most complex cryptography become a decision aid for people who just wanted to know what they were signing. During the FTX collapse, I founded Resilience DAO to support displaced Web3 workers. The pattern is always the same: the technology accelerates until the people closest to it realize the governance structure never left the station. What makes this moment unique is that the engineers are not quitting or leaking. They are asking the state to build the fence.
This also needs to be read against its corporate backdrop. OpenAI spent 2023 in a public boardroom war over exactly this question — the “accelerate vs. protect” divide that nearly killed the company in November of that year. Anthropic has raised billions on the promise that it does safety differently. That employees of both firms signed the same letter signals that safety concerns have transcended corporate strategy. It is now an industry-scale consensus among the people who would have to press the launch button.
Start with the scariest sentence in the letter, because it is doing more work than it appears: “AI research automation.” Most outside observers read this as “AI will replace researchers.” That’s wrong. The signatories are pointing to a self-referential feedback loop: an AI system that can propose, test, and implement improvements to AI systems — including itself — at machine speed. This is not a bigger model. It is a different regime of capability growth, one that does not wait for human comprehension to catch up.
I have audited recursive systems in my DeFi years, and the most dangerous thing you can put on-chain is a loop with no circuit breaker. I remember one lending protocol whose rebalancing function was so tightly coupled to an oracle that a single manipulated price could liquidate the entire pool before a human could broadcast a transaction. We fixed it by adding a pause mechanism. The problem with frontier AI is that no such pause exists by default — and the AI itself is what would rewrite the circuit breaker. This is what “beyond human understanding or control” means in engineering terms: not that the model is smarter than our best scientists, but that its behavior is not causally accessible to them. In crypto, we call that unauditable code. The community always forgives an audit that is slow. It never forgives one that is impossible. Frontier AI safety has crossed into the impossible-audit zone.
And the design complexity is only making the audit harder. I watched Uniswap V4’s hook architecture turn the DEX into programmable Lego a few years ago — elegant, but every added hook is a new attack surface, and more than 90 percent of developers never understood the deeper components they were building with. The same phenomenon appears in AI alignment: every new safety mechanism layered onto a frontier model increases the surface area for failure. Complexity is the silent accelerant of unintended consequences, in both code and cognition.
If you want a mental model for what is happening inside OpenAI and Anthropic, don’t look at the AI safety literature. Look at November 2022. FTX collapsed not because external regulators were asleep — though they were — but because internal governance had been captured by the people with the most to gain from speed. The industry’s response was “never trust, always verify.” We didn’t verify, and the collapse taught us a painful lesson that I now apply to every protocol audit: when the internal checks are controlled by the party being checked, the checks are marketing.
The petition is the AI industry’s admission that this dynamic has taken hold inside its own citadels. These employees did everything right internally first. They raised concerns in writing. They pushed for more safety resources. They watched the deployment calendar win again and again. So they went around the chain to the only external actor with the authority to bind behavior: the state. In crypto we call this “exit to governance” — when an internal fork fails, a community appeals to an external jurisdiction. The OpenAI-Anthropic letter is that appeal, signed in flight.
There is a detail in the letter that deserves attention: the demand for whistleblower protections. Think about what that implies. The signatories believe that present-day AI labs are environments where speaking about catastrophic risk is unsafe for your career. That is an extraordinary internal indictment — and it mirrors the crypto whistleblowers who tried to raise the flag before the 2022 collapse. We now standardize incentive mapping for every protocol’s governance; these labs have moral hazard built into their org charts. When top safety researchers describe their internal process as suffocating the very act of raising a safety concern, no red-team budget in the world will save them. The systemic risk is the reward structure itself.
Here is where I have to betray my own industry’s instincts. As a Web3 evangelist, my instinct is to say: don’t call the government, decentralize. Put the models on-chain, open the weights, let the community audit them. But the signatories have identified something my industry keeps failing to confront: cryptographic transparency is not the same as human safety. Open-sourcing a model with recursive self-improvement capabilities is like publishing the source code for a bug that can rewrite its own compiler. The community can’t review what is fundamentally beyond its comprehension, and the model can improve itself before the forum thread goes live.
And yet the anti-regulatory answer isn’t much better. The petition asks for an “international cooperation mechanism.” That is the IAEA move, applied to silicon. It sounds reasonable, but I watched a version of it fail inside my own sector. During the rollup boom, hundreds of projects rushed to sell dedicated data availability layers when 99 percent of them didn’t generate enough data to fill a single block. The metric existed; the verification didn’t; marketing filled the gap. If governments introduce compute thresholds or safety reporting requirements without a way to verify them, the same pattern will repeat: labs will comply on paper, optimize for the audit checkbox, and find the threshold loophole. Compliance theater is the natural response to unverifiable regulation.
This is where my AI-ethicist work collided with my infrastructure background. At the Human-Centric AI summit I organized in Frankfurt, we spent three days on a single question: are ethical constraints real if nobody can verify them? Our answer, for a subset of use cases, was cryptographic. Zero-knowledge proofs can allow a lab to prove a training run stayed within declared compute parameters — or that an inference-time gate rejected a class of dangerous prompts — without revealing proprietary weights. The technology for verifiable AI claims exists. The transparency that makes it credible does not. And that is a problem no petition can solve by itself; it needs the engineers to keep building, not just asking. I remember the EIP-1559 chaos of 2021, when I drew fee-burning diagrams to calm a community that thought their funds were being deleted. The problem wasn’t the mechanism. It was that nobody could verify it in real time.
Now the politics. The letter asks for international collaboration to control frontier AI, which is the right instinct; models do not respect borders. But the capital-I international framing is where this petition is most likely to get watered down. Treaties are slow; models are not. IAEA-style regimes took decades and worked because nuclear materials are physical and countable. AI is bits, but compute is physical too: GPUs, power, supply chains. The one lever that maps cleanly onto physical control is compute. Training FLOPs thresholds, export restrictions, datacenter licensing. It is no accident that the employees point at frontier development rather than model behavior. Behavior is ethereal; compute is a choke point.
We saw this in crypto as well. Every attempt to regulate token behavior failed. What worked was regulating the fiat on-ramps — the physical interfaces between the digital and the real world. The lesson for AI governance is the same: don’t try to regulate the model; regulate the compute. If Washington and its partners coordinate on compute governance, the effect on the AI industry will mirror the effect of stablecoin regulation on crypto: painful for the move-fast cohort, but a foundation for the institutions that survive the shakeout.
Now let me be the contrarian in the room — and it is uncomfortable, because it is my own tribe that is wrong here. The decentralizer’s instinct is to celebrate any petition that beats against concentrated corporate power. But read carefully: this letter asks the state to regulate a technology, not the community to own it. That is a fundamentally centralized request, and I know how that movie ends. Governments that build fences around frontier technology tend to keep the keys. An AI licensing regime will not be built for the public; it will be built by incumbents who can afford compliance, and it will freeze out the open-source builders who cannot. Regulation could turn AI into the walled garden crypto was supposed to escape.
And yet the alternative is also a fantasy. The open-source absolutists — I was nearly one — say: no gates, let the ecosystem catch up. But you don’t open-source a recursive explosion. Some technologies, like self-replicating code and bioweapons, genuinely need gatekeepers. The uncomfortable truth is that this petition is asking the wrong institution. It defines the problem correctly — internal governance has failed, voluntary commitments have failed — but the solution it reaches for is structurally incapable of the real task. Governments cannot regulate what they cannot verify, and until the verification problem is solved, the petition will produce theater, not safety. I say this as someone who lost friends in the 2022 layoffs to a market that punished speed over safety; I would still rather live with gates I help design than without gates at all. We built cross-chain rails that are technically brilliant yet suffer UX worse than a centralized exchange withdrawal, and the blind spot was never the technology — it was the trust model. That’s the same blind spot staring at us from the AI labs.
Here’s what I’ll remember about this letter a decade from now. Not the policy proposals, and certainly not the regulatory outcome. The signal is in the formation: engineers at rival companies, whose bonuses depend on deployment speed, choosing to form a community around a longer scoreboard. That is the pattern that survives every market cycle. Community is the only chain that cannot be broken.
The next frontier is not who trains the smartest model. It is who builds the most trustworthy one, and the community that proves it. When the models surpass us, we won’t ask whether the state regulated them. We will ask whether trust was earned in ways we could verify. And that’s the only question that has ever mattered.