Governance by design for AI applications: why trust can't be bolted on

Every component was secured. The order still went wrong.

An AI-assisted procurement application gets an urgent request to replace a failed production part. The analyst signs in through the corporate identity provider. The application pulls supplier history, compares quotes, drafts a purchase order, asks for approval in a chat tool, and submits the order to the ERP.

Weeks later, internal audit finds the order was placed for the wrong legal entity, under a master agreement the analyst wasn't allowed to use. The manager had approved a draft amount, but the vendor and price changed after the work resumed. The ERP shows a powerful service account as the actor. The policy log contains a Permit, but not the facts, rule version, conditions or proposal it applied to.

No component broke its own contract. The identity provider authenticated the analyst. Retrieval returned documents it could access. The policy service answered the question it was asked. The approval tool captured a click. The ERP accepted a valid credential.

Why isn't securing each AI component enough?

Because each component checks only the request in front of it. Governance has to hold across the whole chain, from request to consequence, and that's a property of the application, not of any single part.

Enterprise AI governance is usually assembled from capable products: an identity provider at the door, a policy engine on some APIs, an AI gateway, data-loss controls, approval notifications, audit logs and a security platform. Each solves a real problem.

The weakness isn't that they're separate. It's that they don't share one representation of the same piece of work. The user known to the identity system may not be the user seen by retrieval. The approval tool doesn't know which proposal, rules and model version will run when the work resumes. The audit store knows a policy endpoint said Permit, not which conditions were actually met.

There are three levels of assurance, and they're easy to confuse:

An application can pass the first level at every component and still fail the third. More logs and a shared request ID don't close the gap. They show which events belong to the same run. They can't prove each system interpreted the same tenant, legal entity, purpose, delegated authority, approved proposal or condition.

Governance is usually added after the prototype

The usual sequence is understandable. A team proves a model can answer a question or use a tool. Then it adds single sign-on, a policy check, a moderation layer, an approval notification and more logging. The prototype becomes production by accumulation.

It shows in the numbers. EY found that 47% of organizations had bypassed governance to get urgent AI deployments out, and 72% worry they can't trace the lineage behind critical AI decisions. Gartner reports that only 13% believe they have the right agent governance in place.

Secure-by-design guidance argues that protection should be a core product requirement, not a feature added later, and the NIST AI Risk Management Framework treats governance as running through the whole AI lifecycle. Governance by design applies the same idea to how AI applications actually run.

That doesn't mean governance is automatic, free of judgment, or delivered by one product. It means the governed path is the normal path. Weakening a control should require an explicit, reviewable decision, not depend on someone remembering to call a governance service at every new branch.

What does governance by design require?

Five properties: governance is explicit, continuous across every boundary, impossible to skip, bound to the versions that ran, and evidenced by enforcement rather than reconstructed afterward.

• Explicit: the rules and the facts they depend on are represented, not implied.
• Continuous: controls survive service, model, human and asynchronous boundaries.
• Non-bypassable: consequential actions pass through enforcement; there's no side door.
• Version-bound: you can say which workflow, rules, contract, model and approval governed a run.
• Evidenced: enforcement produces a record good enough to defend the result.

Image · governance-by-design-pieces-vs-wholeTwo panels. Left: Identity provider, Retrieval, Policy engine, Approval tool, ERP as separate boxes, each with its own green tick, with visible gaps and question marks between them. Right: the same five boxes threaded by one continuous band labeled Who, For whom, Rules, Approved proposal, Evidence. Caption under left: Secured in pieces. Under right: Governed as one behavior.

Identity has to travel, not just authenticate

Authentication tells you who entered the application. Governed work needs a richer answer at every step: who is acting, on whose behalf, for which tenant and legal entity, for what purpose, with what scope, through which service, against which record and action.

Zero trust architecture asks for granular, per-request, least-privilege decisions instead of implicit trust after login. OAuth token exchange defines separate subject and actor roles for delegation. Neither means forwarding the same broad token everywhere. Each boundary should be able to see both the person who started the work and the component acting now, with a credential limited to its target.

A service account still has a job: it proves which software reached the API. It can't be allowed to erase whose authority is being exercised. Collapsing analyst, agent, legal entity and supplier record into one technical principal makes the call easy to execute and impossible to defend. More on this in AI agent identity and delegated authority.

Enforce policy where consequence happens

A policy that's visible in a builder but optional at run time is documentation. Governance becomes real when policy can intercept behavior before consequence, at every point where information or authority expands:

Outcomes should be richer than allow and deny: permit with conditions, redact, restrict, step up, require approval, route privately, reduce autonomy. Placement follows consequence, not the org chart. The same business rule may appear at several steps, each looking at a different slice of the work. Enforcement can be distributed and cached, but the decision inputs, outcomes, conditions and evidence must mean the same thing everywhere. See runtime policy enforcement for the mechanics.

Can a prompt govern an agent?

No. A prompt can shape what a model tries, but it isn't a dependable boundary. Which tools an agent sees, which records it reads and whether an irreversible action proceeds must be enforced by the runtime.

Declarative rules are only as good as their facts. The runtime needs to know where each fact came from, whether the user asserted it or an authoritative system resolved it, how fresh it is, which tenant it belongs to, and what to do when it's missing. Treating an unavailable attribute as false may be safe for one action and destructive for another, so missing, conflicting and stale inputs need declared behavior, not accidental defaults.

Schemas help too, within limits. A contract can require a payment to carry currency, amount, account, legal entity and operation ID, and reject malformed arguments before anything happens. But a well-formed payload can still be unauthorized. Schema answers "is this well formed?" Policy answers "is this allowed here, now, by this actor, for this purpose?"

Approval is durable state, not a message

Sending a chat message doesn't make a system human-in-the-loop. The notification is a delivery channel. The approval itself must become part of the work's state and bind:

• the exact proposed action and arguments;
• the workflow, rule, context and model versions that matter;
• the evidence the reviewer saw;
• the approver's identity, authority and business scope;
• the decision, conditions, reason and time;
• expiry, escalation and separation-of-duties rules;
• what happens on resume if anything material changed.

Approval is intent, not a blank cheque. A manager who approves a purchase from one supplier under one contract hasn't approved any later payload that happens to stay under the ceiling. Define which fields can change without reapproval and which create a new proposal. Otherwise a person approves one thing and the system executes another, which is exactly what happened in the procurement case. More in human-in-the-loop approval.

Image · governance-by-design-approval-bindingCenter: an Approval card with five attached tags: Proposal fingerprint, Versions, Evidence shown, Approver authority, Expiry. Timeline beneath: Approved, Wait, Vendor and price changed (amber), Resume check, arrow to Back to review rather than Submit to ERP.

Evidence of decisions, not just events

Governance adds a specific evidence requirement: prove what was decided, from which facts, under which rule version, at which step, with which conditions, and what the runtime did with the result. NIST audit controls ask for records of event type, time, source, outcome and identities, while keeping sensitive data minimized.

That isn't a demand to store every prompt and value. Evidence should be risk-based, protected, and able to use references, hashes or masked values where raw content isn't needed. But it must close the loop. A Permit that required two redactions, a second approver and a private model is incomplete until the record shows each condition was applied. And when a decision is indeterminate (an attribute service timed out, two rules conflict), the record should show that and what fallback applied. Quietly turning unknown into Permit is a governance decision hidden inside error handling.

Common anti-patterns

Ten tests for governance by design

1. Every consequential step knows the person, the actor, the tenant, the purpose and the scope.
2. Policy sees typed facts about actor, record, action, context, model and environment.
3. A policy outcome can change or stop the work before any side effect.
4. Rules are versioned, testable, comparable and reproducible.
5. Conditions attached to a Permit are carried out and recorded, not dropped.
6. Every approval is bound to a proposal, evidence, authority, expiry and resume rule.
7. Schemas validate arguments before the write and leave authorization to policy.
8. You can prove which rule version and facts governed any past action.
9. Controls stay consistent across retrieval, models, tools, outputs, storage and replay.
10. Higher-risk actions get stronger controls without forcing every action down the same slow path.

The result: trust as a property of how AI acts

Agentic security guidance keeps returning to the same safeguards: least privilege, limited agency, validation of high-impact actions and human approval where the stakes call for it. Governance by design turns those from advice into how the application behaves. Components can stay separate, and controls can be distributed and tuned to risk, as long as they all serve one shared definition of the work.

Governance shouldn't be something an enterprise reconstructs after an AI system acts. It should be part of how the system is allowed to act at all.