A deadline passed. No announcement followed.
In the registry of state failures, quiet expiration occupies a specific niche. It is not a scandal. It does not trigger an inspector general investigation. It is simply a promise that was made, a date that arrived, and a void where the deliverable should have appeared. The United States government committed to implementing classified benchmarks for evaluating frontier AI models. The commitment had a defined timeline under the architecture established by Executive Order 14110 and the AI Safety Institute's operational mandate. The deadline has crossed. The public record contains no announcement, no progress report, no revised timeline, no white paper, no leaked interagency memo. Just silence.
I have spent more than a decade reading silence in codebases. I know the pattern: a smart contract upgrade that never appears on-chain, a governance proposal that reaches quorum and then stalls, a treasury transaction that vanishes from the mempool. The missing object is never absence. It is a signal, compressed and opaque, but decodable. Silence is a payload, and it carries an entropy profile that can be measured against the expected distribution of state communications.
The code never lies, but the auditors do. I apply the same forensic standard to the US federal government that I apply to every protocol that crosses my desk. The standard is unforgiving. The result is uncomfortable.
The Context: A Laboratory Built for Secrets
The AI Safety Institute sits inside NIST, the National Institute of Standards and Technology, at the US Department of Commerce. It was established in 2024, occupying a narrow parcel of the sprawling administrative infrastructure created by Executive Order 14110. The institute's charter: evaluate frontier AI models before they reach the public, with particular attention to capabilities that could enable cyberattacks, biological warfare, chemical synthesis, or other catastrophic dual-use applications. The model developers—OpenAI, Anthropic, Google DeepMind, and others—signed pre-release testing agreements. The assessments are conducted under protocols the public cannot inspect.
The classified benchmark is the sharpest instrument in AISI's toolkit. It is a secret test. The test items are classified. The scoring rubric is classified. The results are classified. The very existence of the test program is a matter of public record, but every meaningful detail about its operation is sealed. The government calls this operational security. I call it a trust layer with no settlement mechanism.
This is a structural departure from machine learning evaluation tradition. Every major benchmark in the field—MMLU, GSM8K, HumanEval, SWE-bench, MATH—is public. Publicity is not a feature; it is a necessary condition for scientific validity. An evaluation that cannot be independently reproduced is not a measurement. It is an assertion wearing a lab coat.
The United States is not the only jurisdiction building AI governance infrastructure. The European Union has the AI Act, risk-tiered, codified, and subject to parliamentary oversight. China has a registration system for generative AI services with published filing requirements. Neither the EU nor China has produced a classified benchmark program of the kind AISI is building. This is structurally new. Structural novelty is precisely why the missed deadline matters. New institutions have no precedent to guide them, no accumulated trust to protect them. Their first actions set the baseline for every subsequent evaluation.
The US is also the jurisdiction with the most to lose. It hosts the frontier labs. It controls the compute supply chain. Its companies define the global AI market. When the world's dominant AI power builds a secret yardstick and then cannot deliver it on time, the failure is not merely administrative. It is a signal about the reliability of the entire governance experiment.
Core: The Architecture of Silence
Section One: What the Silence Contains
Let me be precise about what the public record actually shows. The classified benchmark program has three conceptually distinct layers: the benchmark sets themselves, the scoring methodology, and the distribution of results. All three layers are classified. The first layer—the test questions—might plausibly be classified for national security reasons. If a benchmark assesses whether a model can design a biological weapon, publishing the test items would effectively publish a how-to manual for bioterrorism. That argument has merit. The second layer—the scoring methodology—is harder to justify. How does the government weight different failure modes? Does a model fail if it scores above a threshold on a dangerous capability, or below a threshold on a safety behavior? The weights encode policy judgments. Hiding them is a democratic and epistemic failure. The third layer—the distribution of results—is the most indefensible of all. If the government has evaluated a model and found it dangerous, the public should know. If the government has evaluated a model and found it safe, the public should know. If the evaluation cannot be shared because the methodology is secret, then the evaluation has no grounding.
This layered opacity creates a specific failure mode, and I have seen it before in on-chain systems. When the audit process is hidden, the audit becomes a checkbox. When the checkbox becomes a compliance artifact, the risk moves elsewhere. In DeFi, we call this audit theater. The government is now building evaluation theater. The classified benchmark exists to establish trust, but trust requires verification. Verification requires transparency. Transparency is absent. The result is an architectural loop that cannot close.
There is a second, subtler problem. The absence of an announcement does not mean the work is not happening. It is entirely possible that the classified benchmark tests are running right now, on schedule, with results flowing to decision-makers through secure channels. The opacity cuts both ways. I cannot verify failure, and I cannot verify success. This is the epistemic trap at the center of classified governance: the citizen is reduced to a spectator who must guess whether the state is competent, and the state has designed a system that guarantees the citizen cannot find out.
In information theory terms, the signal-to-noise ratio of the public record is undefined. There is no signal. There is only the absence of a signal, which is itself information. I have built my career on reading absence: orphaned transaction hashes, unpinned IPFS references, empty audit trails. The pattern here is identical in shape if not in scale.
Section Two: The Reproducibility Principle
I cut my teeth on the 2017 Neo audit crisis. At the time, I was conducting static analysis of Neo's smart contract architecture during its peak ICO phase. I found a critical reentrancy vulnerability in the atomic swap implementation. I documented it with assembly-level proofs. Not a narrative explanation—raw proof, the kind that cannot be argued with. The project leads ignored my report. I published it on my personal blog. Three major exchanges delisted the associated token shortly thereafter.
Why did the exchanges act? Not because they trusted me. They acted because the proof was reproducible. Any engineer with access to the Neo virtual machine could load my assembly trace, replay the attack path, and confirm the vulnerability. The proof carried its own verification mechanism. It did not require faith in the auditor. This is the property that makes technical analysis trustworthy: it can be checked.
Classified benchmarks have the opposite property. They cannot be checked. They cannot be replayed. They cannot be falsified. An evaluation that cannot be falsified has no scientific content. It is a black box that outputs a verdict, and the verdict must be accepted on faith. For an institution designed to protect the public from catastrophic AI risk, building an evaluation system that the public cannot audit is—at minimum—an ironic choice. At maximum, it is a governance catastrophe in waiting.
The problem is not limited to the AI safety community. The reproducibility principle is the foundation of empirical science. When the US government publishes an inflation report, the Bureau of Labor Statistics discloses its methodology. When the FDA approves a drug, the clinical trial data is published. When the Federal Reserve raises rates, the reasoning appears in the meeting minutes. The government's own standards demand transparency for mundane matters like consumer prices and drug efficacy. For the highest-stakes question in technology policy—whether a frontier AI model is safe to release—the government has chosen secrecy.
The asymmetry is indefensible. And the cost of this asymmetry is not abstract. If the classified benchmark produces a false negative—declaring a dangerous model safe—no independent researcher can catch the error. The failure will only become visible after deployment, when the damage is already done. In machine learning, we call this the offline evaluation problem. The government has made it worse by refusing to disclose the evaluation protocol. The first real test of the AI safety apparatus is the test of the apparatus itself, and the apparatus is failing its own deadline.
Section Three: Benchmark Gaming and the MEV of Evaluation
The argument for classified benchmarks is not without merit. I will grant the obvious: if developers know the test set, they can train against it. This contamination problem is well documented. GPT-4's benchmark scores were higher than its actual capabilities suggested, partly because benchmark items leaked into the training corpus. When a developer knows the exact questions, the model becomes a memorization engine rather than a reasoning system. The evaluation is corrupted.
Secrecy is one defense against contamination. But it is a crude defense with crippling collateral damage. And the AI safety community has been building better defenses for years. The DeFi world understands this problem intimately because it faces the same structural challenge in the form of MEV—maximal extractable value. When transaction flow is visible, block producers and arbitrageurs can front-run it. The solution has not been to classify transaction data. The solution is to design mechanisms that make the system robust to partial information.
Commit-reveal protocols are the canonical example. Alice publishes a commitment—a hash of her data—at time zero. At time one, she reveals the data. Anyone can verify that the revealed data matches the commitment. The critical property: the commitment provides cryptographic binding without information leakage. Bob cannot see Alice's data before she reveals it, but he can verify after the fact that Alice did not change her data. This is the exact property that AI benchmark evaluation requires. The test set can be committed to by hash before model evaluation begins, then revealed for public verification after results are locked. Developers cannot adapt to a test set they cannot see. Evaluators cannot retroactively change the test set. The public cannot be fooled about what was actually tested.
The government chose classification instead. Classification is commit-reveal without the reveal. It provides secrecy, but it destroys accountability. In protocol design terms, the government has built a system that is collusion-resistant against model developers but collusion-prone against the public. The power asymmetry is the point.
This is not a technical limitation. The cryptographic tooling for verifiable, secret evaluation exists. Zero-knowledge proofs can demonstrate that a computation was performed correctly without revealing the inputs. Multi-party computation allows multiple parties to jointly evaluate a model on a dataset without any party seeing the full dataset. Functional encryption allows computations on encrypted data. The government, and its contractors, have access to world-class cryptography researchers. The failure to use these tools is a choice, not an inevitability.
The economic consequences of this choice are measurable. Benchmark evaluation is a market. The reward for a model that scores at the frontier on dangerous capabilities is enormous: government contracts, procurement preferences, international influence, public legitimacy. The counterpart is the risk premium applied to models that have not passed the secret test. This is the same information asymmetry that dominates insider-trading regimes. Companies with access to the assessment process—the frontier labs that signed pre-release testing agreements—hold privileged information about the evaluation criteria. Startups without such access must guess. The result is a moat built on opacity.
Section Four: The Open Source Double Bind
The open source model ecosystem is the tension point. Llama, Mistral, Qwen, DeepSeek—these models are distributed as open weights. Anyone can download them, fine-tune them, and deploy them anywhere. An open weights model cannot be evaluated at the point of distribution, because distribution is itself the release. There is no gate, no checkpoint, no moment of regulatory control.
The classified benchmark operates as an ex-ante gate. A developer who wants to prove their model is safe must submit it to a government evaluation process that has never been publicly specified. Open source developers cannot do this without exposing their models to a process they do not understand. The process has no published application procedure, no fee schedule, no timeline, no appeal mechanism. It is a gate that only the privileged can pass through. This has a predictable effect on open source: it drives development overseas and into the shadows. The US loses influence over the behavior of open source developers, while the global community moves to jurisdictions with lighter oversight.
Executive Order 14110 anticipated this problem. Its reporting requirements for large dual-use foundation models were written to capture models with significant weights. The drafters understood that open source distribution uniquely challenges the governance model. The classified benchmark regime makes the problem materially worse because it imposes uncertainty without offering a path to compliance.
I encountered the open source verification problem in a smaller, less consequential form in 2021. I analyzed the Bored Ape Yacht Club collection's on-chain metadata storage. I found that 20% of the PFPs stored critical trait data off-chain via IPFS links that were not pinned. The data could decay. Holders could wake up one morning to find their assets morphed into blank tiles. The trade press dismissed the analysis as pedantry. Custodial institutions quietly cited it as a reason to avoid unverified PFPs for treasury storage. The market absorbed the lesson without acknowledging it.
The parallel to open source AI is uncomfortable. When verification infrastructure is absent, the healthy parts of the ecosystem get punished alongside the unhealthy parts. Institutional participants retreat to what they can verify. The open source model community—the part of the AI ecosystem with the strongest claim to public accountability—loses access to the public market because it cannot pass the secret test.
Section Five: The Verification Market Gap
This brings me to the central observation. What the US government is failing to build, the private sector should build anyway. The AI safety evaluation stack is a market opportunity that has been systematically ignored because the only customer—the government—has chosen to build in-house. The stack is incomplete. There is no public benchmark registry. There is no Merkle-rooted test set with proofs of integrity. There is no open-source evaluation harness that can produce zero-knowledge proofs of model behavior. There is no DAO-governed benchmark curator. The tooling that blockchain has perfected over the past decade—for trustless, verifiable, auditable computation—has not been applied to the highest-stakes verification problem of the decade.
The infrastructure exists. Merkle trees for data integrity. Commit-reveal schemes for test-set secrecy. Zero-knowledge proofs for private evaluation. Verifiable random functions for fair sampling. Token incentives for honest curation. The primitives are proven, battle-tested in billions of dollars of DeFi transactions. What is missing is institutional demand.
The demand is coming. The EU AI Act requires conformity assessments for high-risk AI systems. China requires model registration. The US government is scrambling to build AISI. Every jurisdiction needs evaluation infrastructure, and none of them wants to accept the others' results. The standards will fragment. The market for verified compliance will fragment with them. Somewhere between the EU's paperwork and the US's classified black box, there is room for a verification layer that produces actually checkable claims.
I have built predictive models of incentive structures for a decade. My 2020 analysis of Curve Finance's veTokenomics predicted that the IRV implementation would create arbitrage opportunities for insiders. The exploit came six months later. The loss was $1.5 million. I did not celebrate. I documented. The lesson was that incentive modeling works when the model includes every relevant actor. The US government's classified benchmark program has a predictable incentive structure: the agency wants to minimize political accountability, the model developers want high scores on unknown criteria, the public receives nothing. There is a systemic misalignment, and it is unpriced.
The pricing of this misalignment is visible in the valuation of AI companies. Publicly traded AI equities trade at valuations that implicitly assume safe, orderly, measurable deployment of frontier models. If the classified benchmark regime produces a false negative, if a dangerous model slips through the secret test, the market repricing will be dramatic. The risk is uninsurable because the evaluation is unauditable. This is a systemic risk to the AI trade.
Section Six: State Capacity Audit
The deadline slippage is a resource allocation problem. AISI is a new institution with a staffing pipeline that cannot compete with private-sector AI labs. The federal hiring process takes months. Clearance processes take longer. Model development cycles run at 18 months. Federal hiring cycles run at 24. The mismatch is structural.
I read the missed deadline as a capacity constraint rather than a political event. The people building the classified benchmark are not idle. They are fighting an interagency dispute over classification levels, security clearances, and the jurisdictional boundaries between NIST, the Department of Commerce, the White House Office of Science and Technology Policy, and the intelligence community. In protocol terms, this is committee deadlock. The committee cannot reach consensus, so the deliverable does not ship.
The government's weakness is not malicious. It is structural. Bureaucracies are poorly designed for evaluating technology that changes every quarter. The AI safety evaluation problem requires domain expertise that the government has not yet acquired. The people most qualified to build the benchmark are the same people being evaluated. Conflict of interest is baked into the staffing model.
The result is an execution gap. The US government has the ambition to evaluate frontier AI models but not the institutional capacity to do so on time. This gap is information. It tells us the government will rely on the frontier labs for substantive expertise, which means the labs will influence the evaluation criteria. The fox will have a significant hand in designing the henhouse's security system.
Contrarian: What the Secrecy Advocates Get Right
I have built a career on distrust of centralized verification. It would be intellectually dishonest to ignore the legitimate arguments for classified benchmarks.
National security requirements are real. A benchmark that tests whether a model can synthesize a novel biological weapon cannot be published in full, because publication itself would constitute a biosecurity threat. The operational security argument has merit. The government's obligation to evaluate threats without publishing threat recipes is genuine.
Benchmark contamination is also real. Secrecy is crude, but it works. A model developer who cannot see the test set cannot memorize the answers. The performance degradation that comes with adaptation to unknown criteria is a feature, not a bug. The classified benchmark will produce a more honest measure of generalization capability than any public benchmark can.
And some opacity is necessary for operational security. The government is not a public blockchain. It has legitimate reasons to withhold information about its assessment processes, particularly when adversaries might use that information to evade detection. The cold calculus of state security does not map cleanly onto the open-source ethos of the blockchain community.
But the crypto critique stands. The middle path exists. Zero-knowledge proofs allow verification of a computation without revealing the underlying data. The same primitive that lets a blockchain prove a transaction is valid without revealing its contents could let the government prove it has evaluated a model against specific criteria without revealing the criteria themselves. Regulators are not taking this path. The failure is not secrecy. The failure is secrecy as the only tool.
There is a deeper blind spot in the defenders' position. They assume that government secrecy is efficient. History is replete with examples to the contrary. The classified verdict might be confident and wrong. The absence of an independent check on the evaluation process converts every error into a catastrophe, because there is no mechanism for correction.
Takeaway: The Settlement Layer Is Missing
The AI safety evaluation stack will be built. The question is whether it is built on the public settlement layer—with verifiable commitments, auditable results, and the ability for anyone to replay the computation—or in the dark, where someone's assertion is all we get.
If the benchmark cannot be audited, the safety claim is fiction. If the deadline passes with no acknowledgment, the process is fiction. If the market prices classified evaluation as a black box, it will assign a risk premium that no one can verify.
Trust is a vulnerability with a capital T. The entire blockchain stack was invented to remove the capital T. The United States government is building its AI safety infrastructure on the exact trust model that the technology was created to avoid. This is the deepest irony of the whole enterprise: the cryptographic toolkit for building auditable governance exists, and the government will not use it.
The next step is not to demand that the government open its secrets. It is to build the alternative infrastructure—verifiable, public, provable AI evaluation—and let the market choose. The code never lies. It is the people who keep it in the dark who need watching.