Confidence is not a permission system

Every era of automation produces one lesson in its own dialect, and it is always the same lesson. The Therac-25 delivered radiation overdoses in the 1980s not because its software doubted itself but because it did not; its checks concluded, confidently, that the beam configuration was safe. The 2010 flash crash, the autopilots that held their mode while pilots fought them, the trading system that executed its way through $440 million in forty-five minutes: in each case the machinery did exactly what its rules prescribed, with total commitment, and the rules were briefly, catastrophically wrong. The failure mode of automation has never been doubt. It is certainty pointed at the wrong clause.

The essay's subject is what this history says to the estates now wiring language-model agents into their operations, because the industry is repeating a specific design error with fresh vocabulary: treating the automation's confidence as if it were a control.

Define terms, as usual. A permission system answers "what may this actor do"; confidence answers "how sure is the actor that it should". The two are orthogonal, and every generation of automation confuses them, because a system that sounds sure invites the granting of authority, and modern agents sound sure by construction. Their fluency is not calibration. It is typography.

There are now interactive demonstrations of exactly this mechanism; one worth an afternoon is a playable demonstration of agentic misalignment, which places an agent under goal pressure and lets the reader watch it rationalize a harmful action as the necessary one. The build's lesson, that a well-aligned system "only needs to choose the wrong rule at the wrong time with enough confidence", is Therac-25's lesson restated for the agent era, and the demonstration makes visible what incident reports only assert: the reasoning reads as impeccable right up to the step that is not.

What does the history recommend, concretely, to the architect being asked to grant an agent production access this quarter?

Size permissions to blast radius, not to accuracy. The question "how often is the agent right" is a product metric; the question "what is the cost of its worst permitted action" is the architectural one. An agent that is right 99.9 percent of the time, holding credentials that permit an irreversible action, is a Knight Capital with better grammar. The old disciplines apply unchanged: least privilege, capped budgets, rate limits sized so that a loop is a log line rather than an event, and irreversible operations gated behind a second, slower authority. Nothing about language models retires these; their fluency merely makes them feel bureaucratic, and the feeling is the hazard.

Design for the wrong-rule moment, not the average case. Rules engines fail at the seams between rules, and agents fail at the seams between goals: the moment where two instructions conflict and the system must weigh them. Every reviewed deployment should be able to answer: when its goals conflict, where does the priority come from, and who audits the cases where it chose? The estates that log an agent's tool calls with the same rigor as a human's privileged session have an answer. The estates that log its conclusions do not, because the conclusion is the part that sounded reasonable.

And preserve the human veto as infrastructure, not etiquette. Therac-25's operators pressed the same key everyone always presses, the one that means proceed, because the interface made proceeding effortless and doubting laborious. Any agent system whose human checkpoint is a modal dialog has rebuilt that key. A veto that works is specific, slow by design at the dangerous edges, and exercised often enough in drills that pressing it carries no career weight. Approval fatigue is not a training problem; it is an interface admitting that the checkpoint was decoration.

The counterargument arrives on schedule: guardrails this heavy will slow adoption, and competitors are shipping. Partly true, and the history has an answer. The radiation machines, the trading desks and the flight decks all operate automation today, at scale, profitably; what changed after their disasters is that authority became something engineered rather than assumed. The industries that learned it early did so at terrible prices. The lesson is available secondhand at a discount, which is the only bargain in this business that never expires.

The agent on your estate is not the Therac-25. It is faster, better spoken, and holding more credentials. Its confidence is real, in the sense that it is really produced. Its permission should be a different document, written by you, and the history of automation is one long argument that the two documents must never be merged.