AI agents are a new software abstraction: why code alone can't explain them

The refund was right. Nobody could say why.

A support agent recommends a refund. A representative approves it, the payment service sends the money, and the case closes. Every service reports success.

Months later, an auditor asks simple questions. Which customer records went into the model's context? Which policy version allowed the refund? Which model and prompt produced the recommendation? Did the action run with the representative's authority or a broad service credential? Did the approver see the same recommendation that was executed? Which API arguments moved the money?

The team has fragments: application logs, a trace, a prompt record, a policy decision, an approval entry, an API response, database history. What it doesn't have is one place where those fragments were ever connected.

This isn't a story about a bad answer or a failed service. It's about a system whose real behavior was never written down in one place.

What changed in the smallest unit of software?

The smallest unit used to be a function that does what it's told. With AI it's a step that decides: it reads context, weighs options and picks what to do next while it runs.

A function has a known contract. Given the same version and inputs, it follows the intended path, and when it breaks you look for the line that failed. A deciding step is different. Its output depends on what it was shown, which model answered, which prompt framed the question and which tools were available at that moment. Two near-identical cases can take different paths, and both can be reasonable.

That moves the important decisions from development time to runtime. A developer no longer fixes every path in advance. The path is chosen when the case runs. So the questions you used to answer by reading code now have to be answered three times: at design time (what is this step allowed to do?), at run time (what is it doing, and should it?), and after the run (what did it do, and why?).

An AI application is not a model call

Most AI conversations start with the model: which one, how well it reasons, what it costs. Those questions matter, but a production application does far more than call a model.

It retrieves records and documents, assembles context, classifies the request, computes known facts, picks a model, renders a prompt, calls tools, validates the output, checks policy, waits for a person, writes to a system of record and keeps evidence. Different teams own each piece. Each piece is versioned and deployed on its own schedule. Different security rules govern the data each one touches.

Research has said this for years. Production machine-learning work showed the model is often a small part of a larger system whose dependencies and configuration create lasting technical debt. NIST's AI Risk Management Framework treats AI as a socio-technical system spanning people, data, oversight and change, not as an isolated model. Retrieval makes the chosen documents part of the behavior. Reasoning-and-acting patterns make tool results part of the path. Tool-using models make the choice of tool and its arguments another thing that varies at runtime.

The result isn't "software plus an LLM." It's a system whose behavior comes from relationships spread across many artifacts and runtime conditions.

Why can't the code repository explain AI behavior?

Because much of what determines an AI action lives outside the code: retrieved documents, prompts, model routes, policy versions, delegated identity and approval state all change without a code deploy.

Code is still the best way to implement algorithms, adapters, validation and specialized capabilities. Nothing here argues otherwise. The limit is narrower: the repository can't tell you everything that produced a particular action.

One release can behave differently when any of these change:

• the documents in the retrieval index, or their permissions;
• the data a calculation reads;
• the prompt template or the rule for assembling context;
• the model, provider or routing threshold;
• the policy or business rule;
• the user, the agent acting for them, the tenant or the scope;
• the set of tools available, or an API contract;
• the approval state, or the version resumed after approval.

You can reference all of these from code. That doesn't make the code the record of behavior if those things change independently or live in other systems. So it helps to separate representations that usually get blurred together:

A trace deserves a special note. Trace standards are good at correlating operations across services, and newer conventions add model and tool details. But a trace records what was observed. It isn't the definition of what was permitted, which approval was required or which evidence the application promised to keep. Telemetry can fill in a definition. It can't replace one.

The hidden workflow already exists

Every AI application already has steps and connections. The steps are users, agents, services, models, calculations, data sources, policies, approvals, APIs and records. The connections are control flow, data flow, context inclusion, delegation, policy checks, version dependencies, approvals, retries and evidence links.

Today those connections are scattered across code, configuration, prompt templates, API definitions, queries, tokens, policy rules, queue messages, model gateways, approval tools and observability platforms. Engineers rebuild the picture in their heads when debugging. Auditors rebuild it from evidence after an incident. Security teams infer it from diagrams and service permissions. Business owners see a few process stages.

Drawing the workflow doesn't create it. It shows that it was always there. And calling it a workflow doesn't mean one database or one physical server. The steps can run in different services, clouds and data centers. The definition stays logically unified while the work stays distributed.

A concrete failure shows why this matters. A tenant-scoped support app filters its database query correctly to Tenant A. Its document search uses an index that doesn't keep the same scope, and a Tenant B document slips into the context. The model writes a fluent answer and every service reports success. No component is broken. The failure lives in the relationship between identity, tenant scope, the search index and the context. Only a representation of that relationship can catch it.

Why does AI need a runtime the customer owns?

Because the work crosses systems nobody owns end to end. Identity, policy, approvals, durability and evidence have to live in one runtime you control, not in each tool or vendor along the way.

A refund touches a CRM, a model provider, a policy service, an approval tool and a payment system. Each has its own logs and its own idea of who the caller is. None of them sees the whole action. If governance lives inside each component, the gaps between components are where things go wrong, and nobody is accountable for the gaps.

So the controls have to sit in the place the work actually flows through: a runtime that checks who is acting and for whom, enforces policy before and after each step, pauses for a person when the rules say so, survives a crash, and keeps one record of the decision. And because that runtime holds your identities, policies and evidence, you need to own it, in your data center or your own cloud account.

There's a historical echo here. Economic historian Paul David showed in 1990 that electrification paid off only when factories were redesigned around small motors driving each machine, not when the dynamo simply replaced the steam engine on the old central shaft. Bolting a model onto yesterday's application shape is the dynamo on the shaft. The payoff comes from redesigning around the new unit.

What makes a workflow definition governable

A useful workflow definition has six properties.

1. First-class. It's an artifact with an owner, permissions, validation, history and deployment status, not a diagram rebuilt from screenshots after the fact.
2. Versioned. Each run is tied to the exact definition and the versions of the prompts, models, policies, schemas and tools it used. "The code didn't change" is no longer proof that the application didn't change.
3. Executable. The definition is what runs. It may be compiled into faster forms, but the reviewed version and the running version are the same thing.
4. Contract-aware. Inputs, outputs and mappings are typed. A model stays probabilistic, but its boundary is checked, so failures can be classified.
5. Identity- and policy-aware. It shows who is acting, for whom, in what scope, where policy is checked and what result gates the next step, even when the identity provider and policy engine are separate systems.
6. Evidence-connected. It defines where each run's evidence belongs: context sources, model and prompt version, policy decision, approval, tool arguments and the confirmed result.

One definition, many views

A shared definition doesn't mean one screen for everyone. Developers look at schemas, mappings and payloads. Security looks at identity, data classes, policy checks and credentials. Auditors look at versions, approvals and evidence. Operations looks at state, latency, retries and cost. Business owners look at stages, exceptions and who approves what.

Today each group reviews its own artifact: source code, a threat model, records, a process diagram, dashboards. Each view can look coherent while gaps open between them. When every view points back to the same versioned steps and the same run evidence, those gaps close. Permissions can limit what each role sees, but nobody is reading a separate story.

What this isn't

• Not an architecture diagram. Diagrams go stale unless they're tied to what runs.
• Not an agent framework's graph. That graph decides what the agent tries. The governed workflow also carries contracts, identity, policy, approvals, durable state and side effects. (See workflow graphs vs agent graphs.)
• Not a trace viewer. Traces are evidence of what ran, not a definition of what's allowed.
• Not a renamed workflow engine. The test is whether context, authority, policy and evidence are built in or bolted on.
• Not one monolith. The definition is unified; the work can run anywhere.

Common anti-patterns

Seven rules for building around deciding steps

1. Treat every model or agent call as a step that decides, not a function that obeys.
2. Keep code for capabilities; keep behavior in a versioned, executable workflow definition.
3. Make each step's inputs and outputs typed, even when the step is probabilistic.
4. Record who acts, for whom and in what scope at every step.
5. Check policy before and after each step, not only at the front door.
6. Tie every run to the exact versions it used, and attach its evidence to the same definition.
7. Run all of this in a runtime you own, because no single system sees the whole action.

The result: complexity you can see

AI doesn't need less engineering. It needs a better representation of what the engineering built. A governed workflow definition doesn't remove complexity; it turns scattered, implicit relationships into something explicit, versioned and inspectable at design time, at run time and afterward.

The strongest reason to adopt it isn't that diagrams are easier to read. It's that you can't govern material AI behavior when its real definition is spread across code, prompts, model routes, indexes, policies, identity systems, approval records and tool integrations. The hidden workflow already exists. The choice is whether to leave it hidden. For the evidence side of the same problem, see audit trails: logs vs evidence.