The Black Box That Governance Built
There is a passage in corporate governance history that does not appear in any textbook, because it is not a single event. It is a pattern — repeated across industries, across decades, across crises that were later described as unprecedented but were, in fact, predictable. The pattern goes like this: an organization adopts a system it does not fully understand, the system produces outcomes that serve the organization’s short-term interests, and for a period of time that ranges from months to years, the gap between what the organization knows about the system and what the system is actually doing is invisible. Until it is not.
We saw this pattern in the financial crisis of 2008, when synthetic collateralized debt obligations became so complex that the boards approving their use could not explain how they worked or what their failure modes were. We saw it in the pharmaceutical industry’s relationship with opioid distribution algorithms that optimized for volume without flagging concentration patterns. We saw it, repeatedly, in the history of information technology: systems deployed at scale before governance structures existed to oversee them, followed by consequences that were described as unforeseen and were, in every case, the predictable result of governing at a lower level of sophistication than the system being deployed.
We are in that pattern again. And this time, the system is artificial intelligence.
The institutional instinct that produced this pattern is not malice. It is something more dangerous: the reasonable-sounding belief that operational complexity belongs to operators, and that governance bodies can fulfill their responsibilities through outcome monitoring rather than mechanism comprehension. This belief has a name in corporate governance theory — it is called the “black box tolerance” — and for most of the postwar era, it was defensible. Boards could not be expected to understand the chemistry behind every pharmaceutical their company manufactured, or the engineering behind every product their factories produced. They hired experts. They set standards. They monitored outcomes. That was enough.
The AI era has ended the era in which black box tolerance is defensible, for a structural reason that has nothing to do with AI’s complexity and everything to do with AI’s autonomy.
The systems that preceded AI operated within boundaries that governance structures could observe. A pricing error produced a mispriced product. A distribution error produced a supply chain disruption. These outcomes were visible, traceable, and correctable through normal management processes. The board did not need to understand the mechanism to observe the outcome.
AI systems are different because they make decisions at a speed, scale, and degree of autonomy that breaks the feedback loop between decision and observable outcome. A pricing algorithm that has been trained on shared competitor data and optimized for margin does not produce a visible pricing error — it produces a pricing pattern that looks like normal market behavior until a regulator’s data science team reconstructs it. A disclosure that describes “proprietary AI capabilities” without quantifying the underlying third-party dependency does not look like a material omission until the SEC’s comment letter arrives. A board that lacks the technical fluency to interrogate management on AI risk does not feel the absence of that fluency until the question is asked in a deposition.
This is what makes the current moment structurally different from previous technology governance failures. The previous failures were failures of oversight. The current failure is a failure of comprehension — and comprehension cannot be delegated.
The MIT Sloan Management Review’s research finding that companies with technical directors on their boards command a 15% valuation premium is not a coincidence. The market is pricing something real: the difference between a board that can ask the right questions about consequential systems and a board that cannot. The 15% is not compensation for technical knowledge. It is compensation for the reduction in the probability of the kind of governance failure that produces forced restatements, FTC enforcement actions, and EU market lockouts.
What the governance literature has not yet fully grappled with is the implication for board composition criteria. The current standard for board qualification is built around financial expertise, industry experience, and functional leadership — the accumulated wisdom of running the kind of organization whose risks governance structures were designed to oversee. That standard was calibrated for a world in which consequential organizational decisions were made by humans who could be questioned, whose reasoning could be examined, and whose choices left a trail that audit and compliance functions could follow.
The algorithmic organization does not work that way. Its most consequential decisions are made at machine speed, in conditions of genuine operational complexity, by systems whose reasoning is often not reconstructible after the fact. Governing that organization requires a fundamentally different kind of board sophistication — not replacing financial and industry expertise, but adding to it a form of technical comprehension that the traditional board selection process was not designed to produce.
The leaders who will build the organizations that outlast this moment are the ones who are willing to restructure their governance architecture for a world in which the most consequential systems they deploy are also the hardest ones to see.
That requires something rare in institutional life: the willingness to acknowledge that the governance structures that served the organization well until recently are no longer sufficient, and to change them before the consequences of insufficiency become visible.
In every previous iteration of this pattern, that change came after the crisis. The question this moment places before serious leaders is whether it can come before.
The answer is not technical. It is a question of institutional character — whether the organization values the comprehension it does not yet have more than the comfort of the delegation structures it already possesses.
That is the question worth carrying forward.