Who Can Stop an AI Lab? Right Now, No One
Mapping authority in frontier AI governance and predicting what comes next.
The most thought-provoking AI governance framework I’ve read lately is one I couldn’t accept, which is what made it so useful: assent teaches you nothing about your own foundations; resistance maps them.
In “A Framework for the Private Governance of Frontier Artificial Intelligence” (Ball, 2025), the USG oversees private certification bodies that certify AI labs. In this hybrid model, developer participation is voluntary, which means labs “opt in” to certification and are shielded from tort liability for customer misuse of their models. That clause deserves a second reading: shielded from tort liability. The certifying bodies are also approved and overseen by the USG, which concentrates a high degree of power in the government. Despite the many ways certification bodies can fail, or be incentivized, pressured, or corrupted by actors with other agendas, Ball’s essay contains a safeguard worth pulling forward: pre-deployment government testing for national security risks. Why? Because it’s already in place in other arenas, and historically, it works. Where I part ways with Ball is the opt-in itself: AI governance frameworks cannot rest on voluntary foundations. Opting in is a peacetime design, and the race to superintelligence is anything but peaceful. History has run this experiment before. Let’s look at the races.
From Sputnik to AI: How US-China Competition Builds Institutions
In October 1957 the Soviet Union put Sputnik into orbit, and the United States discovered overnight that it was behind. The USG did not respond with a pledge or a summit; it created ARPA, the agency that became DARPA, carrying a mission that fits in one line: make sure the United States is never surprised by technology again. The same shock rebuilt American science funding and education, and the atomic age built the Atomic Energy Commission before it. ARPA also funded ARPANET, the network that became the internet.
AI is the modern-day sequel to Cold War containment. The competition between the United States and China over AI is the closest thing this century has produced to a Sputnik moment, and it is already being used as an argument against governance: we cannot afford oversight, the reasoning goes, because they will not slow down. History says the opposite. Competitive shock is precisely when America builds institutions, because competition is what raises the price of catastrophe high enough to pay for them.
The nuclear track of that history also left us its hardest lesson. For twenty-eight years the AEC both promoted nuclear power and policed it, and the arrangement failed so visibly that in 1974 Congress tore the agency in half, separating the promoters from the regulators and creating the Nuclear Regulatory Commission. Never house promotion and policing in the same body. The US government’s own AI evaluation center now carries an explicit innovation-promotion mandate alongside its testing role. In this light, it is easy to argue that we have rebuilt the AEC.
The AI Council of Governors: A New Model for Frontier AI Oversight
My proposal is for an AI Council of Governors (AICoG); conceived like ARPA (so, prompted by global competition), structured like the NRC, staffed like DARPA, and monitored/warned by the AEC.
The AI Council of Governors (AICoG) would take it’s skeleton from the NRC: an independent commission whose members serve fixed, staggered terms with partisan balance caps, whose budget is recovered from the industry it licenses, and whose inspectors hold unannounced access as a condition of the license. AICoG could be staffed like DARPA; term-limited technical program managers with real hiring authority, so the Council is staffed by people who can interpret weights, training, and so forth. Lastly, from the AEC it takes the prohibition: the Council will regulate.
Six Design Principles for Frontier AI Regulation
1) Independence by statute. Council members serve fixed, staggered terms and are appointed through distributed channels, so no single administration can stack it and no single election can capture it. Members are barred from AI firm equity and board seats.
2) Mandatory, entity-based oversight. All Frontier AI labs must participate. Safety is not voluntary.
3) Auditors with no stake in the answer. The Council’s auditors come from academic AI research labs with no industry ties, benefiting from participation in the science rather than from the firms they examine. The institutional vehicle already exists: the FFRDC model that gives us MITRE, RAND, and the national labs, nonprofit technical organizations that work only for the government and are barred by federal acquisition regulation from competing with industry.
4) Continuous, not annual. Auditing is continuous, random, and automated, not a yearly ritual the firm can prepare for. This sounds radical until you notice it already exists: the NRC stations resident inspectors inside every operating reactor, with offices in the plant, and the largest banks live with embedded federal supervision teams year-round. Continuous oversight of private infrastructure is not a new idea. It is how we already treat everything else that can fail catastrophically.
5) Enforcement through compute. Noncompliance carries an immediate consequence: access to compute. This is the one place on today’s map where enforcement already works, because compute is physical, countable, and has a chokepoint.
6) Coordination as an obligation. Frontier firms must coordinate on safety through the Council, which acts as the required intermediary and resolves the antitrust barrier that labs currently cite as a reason they cannot share safety information. Coordination is scored annually. The scoring is designed to resist Goodhart’s law, the results are presented publicly, and firms can question and appeal.
One can also see parallels with pharma; the Council treats frontier AI similar to pharmaceuticals rather than software. It preempts state law so the highest risks are governed federally. And it faces a five-year congressional reauthorization, so the institution itself must periodically justify its existence to the public’s representatives. Internationally, the same structure repeats: an allied council of national regulators coordinating standards and enforcing them through shared control of compute exports, so no lab escapes oversight by relocating, and every country has a seat at the table.
Put the proposal on the map and something surprising happens: almost nothing moves. The same labs, the same evaluators, the same public. What changes is the lines. Dashed becomes solid. Invited becomes mandatory. Point-in-time becomes continuous. And the broken arrow, for the first time, connects to something. The redesign is not a new bureaucracy stapled onto the system. It is the same system with authority flowing through it.
The Price of AI Oversight
Staggered terms, distributed appointments, external auditors, and reauthorization answer the “who oversees the overseers” question only partially, and I do not believe any design on paper answers it completely. At first, mandatory oversight may slow American development relative to less regulated countries. And compute-based enforcement requires visibility into private infrastructure that firms will fight, but is doable since data centers require a physical location and chips can be traceable. Take for example, this idea of chip treatment from Superintelligence Strategy:
“By treating AI chips as they would fissile material—cataloging each unit, supervising its destination, and guarding against unauthorized diversion—states can deny terrorists or other rogue actors the raw power to train their own weaponizable AI.”
During an intelligence explosion, power is going to concentrate. The only real choice is whether it concentrates inside AI labs, with the USG, or inside an institution deliberately built to hold it, constrain it, and answer for it, one of which that is insulated from USG pressures and incentives, but also reports to the USG and utilizes federal mandates. In sum, the AI Council of Governors acts as a Stewardship, embedded in the government structure like a central bank, but independent. (Hendrycks, et al., 2025)