Companies are beginning to give software authority that used to belong to people. A system screens applicants. Another recommends pay changes. A third drafts a termination letter and sends it into a workflow that may be too efficient for its own good. Some of these systems are ordinary automation. Others are agents: software authorized to pursue an objective through a series of decisions or actions, sometimes without approval at every step.
The distinction matters. So does the consequence. A writing assistant and a system that can alter payroll do not belong on the same risk register. The board does not need a tour of every clever tool in the company. It needs evidence that management knows which systems can affect people, money, rights, safety, or the company itself.
The board already owns oversight of those consequences. Management owns the machinery that controls them. A machine doing the work does not change either responsibility.
Governance is ordinary work
Governance decides who may decide what, within which limits, and how the organization will discover that a limit was crossed. It also decides who cleans up afterward. That last part is usually missing from the diagram.
Money provides the familiar example. The board approves budgets and risk limits. Managers spend within them. Large payments require another signature. The person who creates a vendor cannot also pay it. Someone outside the transaction reconciles the record. Accounting is an information system for governance, and it is boring on purpose.
The analogy has limits. Financial processes change. Estimates move. Fraud adapts. Still, the important changes are usually visible and controlled. AI can change because the model changed, the prompt changed, the data changed, a tool was added, a permission widened, or users found a behavior nobody anticipated. Sometimes nothing in the written procedure changes at all.
The Linda problem
A surprising amount of governance lives outside the policy manual. Some of it is built into software. Some sits in meeting habits. The rest lives in the head of Linda, who started in 1998 and remembers why the innocent-looking exception is not innocent.
This arrangement is less foolish than it sounds. Writing every rule down costs money, and rules go stale. People compensate. They notice that something feels unusual. They stop. They ask Linda.
An AI system can be instructed to ask Linda. The hard part is getting it to recognize when Linda needs to be asked. A system operating inside an incomplete rulebook may proceed with great confidence, at considerable speed, and with no sense that it has crossed from the routine into the strange. Scale turns a small misunderstanding into an operating method.
Access is not authority
A key to the storeroom does not authorize its holder to empty the shelves. Organizations understand this about employees and routinely forget it about software. If a program can reach payroll, many systems treat that access as permission to change payroll.
The authorization model should separate five activities: seeing information, sharing it, recommending an action, executing the action, and delegating authority to another system. The list is not sacred. It is useful because it exposes permissions that disappear inside a broad label such as access.
Delegation deserves special attention. An agent that can assign work to another agent has not merely gained another tool. It can create a chain of action whose participants, evidence, and stopping points may be hard to reconstruct. The company should know where authority began, how it traveled, and where it ended.
One action can cross several boundaries
Consider an employee wellness chatbot. An employee says that they have had cancer and are not sleeping. A poor response could create a privacy incident, offer unsafe clinical guidance, miss an escalation condition, and produce an employee-relations problem. Different control functions will see different failures. They may discover the incident late, count it several times, or lose it in the space between departments.
That is why AI governance cannot be divided into separate security, legal, HR, quality, and ethics projects with no common owner. The organization needs one record of the system, one account of the decision, and a defined way to resolve conflicts among reviewers. Otherwise, everyone owns a fragment and nobody owns the result.
What management has to build
The starting point is an inventory of consequential uses. It should describe the intended purpose, prohibited uses, affected population, data, vendor and model dependencies, authority granted, accountable executive, testing evidence, known limits, and conditions that require a handoff. It should also identify who may suspend the system and what happens after suspension.
For each consequential system, management should establish:
A named business owner who is accountable for the outcome, not merely the installation.
Separate permissions to see, share, recommend, execute, and delegate.
Performance thresholds, subgroup checks, uncertainty limits, and events that are never acceptable.
Predefined conditions for escalation, human review, appeal, and remediation.
A record of significant actions, approvals, model changes, data changes, and delegated work.
Independent review with authority, access to evidence, and a clear reporting line.
A containment and recovery plan that can stop transactions, revoke credentials, preserve evidence, notify affected people, and restore service safely.
Regular certification that the inventory is complete and deployed systems match what was approved and tested.
This is the accounting model pointed forward. It records authority before the action, watches behavior while the system operates, and reconciles the result afterward.
The off switch digression
People like off switches. They are visible. They fit nicely in a presentation. Unfortunately, an off switch is often a fire alarm pretending to be a fire department.
Stopping the interface may leave queued transactions running. Copies of data may remain elsewhere. A vendor may still be processing requests. Another agent may retain delegated credentials. The real requirement is containment: the ability to stop new actions, find work already in motion, preserve the evidence, limit further damage, and recover without quietly switching the same problem back on Tuesday morning.
Build authority in stages
Start with one decision that matters, such as a pay change. Write down what the system may do and who owns the result. Confirm that the approved configuration, the tested configuration, and the deployed configuration are the same thing. This is less common than the policy binder suggests.
Let the system observe. Then let it recommend. Allow action with approval only after the evidence supports that step. Independent action comes later, if it comes at all. Each increase in authority should be earned by results and reversible when the results deteriorate.
Testing carries much of the weight, but volume is not coverage. A million variations of an easy case are still an easy test. The program needs representative decisions, repeated trials, rare but severe failures, subgroup analysis, adversarial use, changing conditions, and regression tests after every material update. It also needs tests of the reviewers. Automation bias is well documented. A human who never says no is part of the interface, not a control.
What the board should receive
The board should not operate the company’s AI systems. It should set the appetite for consequential uses, require a credible control system, and test management’s claims through independent assurance. The reporting should be short enough to read and concrete enough to interrogate.
Before the next meeting, ask for the inventory. Ask which systems recommend and which can act. Ask what changed since the last report. Ask which limits were crossed, which systems were stopped, and whether the people assigned to supervise them ever overruled them.
Before the next insurance renewal, read the exclusions and endorsements across D&O, cyber, technology errors and omissions, employment practices, and other relevant coverage. AI risk is being allocated more explicitly in insurance contracts. Coverage depends on the policy language and the theory of the claim. There is no useful version of the sentence, ‘We assumed it was covered.’
Before the quarter closes, require accountable executives to certify that the inventory is complete, deployed authority matches approved authority, testing remains current, and material incidents reached the right reviewers.
If those answers do not exist, the first governance report has already been delivered.
Sources
National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework and Generative AI Profile. https://www.nist.gov/itl/ai-risk-management-framework
European Commission. AI Act implementation timeline and compliance resources. https://ai-act-service-desk.ec.europa.eu/en/ai-act
Colorado General Assembly. SB24 205 Consumer Protections for Artificial Intelligence. https://leg.colorado.gov/bills/sb24-205
Goddard K Roudsari A and Wyatt J C. Automation bias a systematic review of frequency effect mediators and mitigators. https://pmc.ncbi.nlm.nih.gov/articles/PMC3240751/
Fenwick. The End of Silent AI Emerging AI Exclusions Coverage Fragmentation and Practical Implications for Policyholders. https://www.fenwick.com/insights/publications/the-end-of-silent-ai-emerging-ai-exclusions-coverage-fragmentation-and-practical-implications-for-policyholders
Absolutely fantastic breakdown! Thank you for sharing.