AI applications are systems, not model wrappers: what the model can't own

The answer was right. The credit went to the wrong account.

A software company uses an AI assistant to review service-credit requests before renewals. A representative opens the parent account, Asteron Holdings, then switches to a subsidiary, Asteron Europe, while the assistant's conversation stays open. She asks whether the subsidiary qualifies for a $24,000 credit after a documented outage.

The model does well. It finds the subsidiary's contract, the relevant service-level clause and the evidence, calculates the allowed credit and returns a valid structured proposal. The recommendation is correct for Asteron Europe.

The tool call isn't. The CRM connector takes the amount and rationale from the model but pulls the account ID from conversation state created when the parent account was first opened. The screen shows the subsidiary. The hidden context still points to the parent. A broad service credential can update both, so the CRM accepts the request and returns success. The assistant confirms the credit was applied. It was, to the wrong legal entity.

Every component checks out: right contract, valid JSON, valid credential, CRM did what it was told, success message shown, no failed step in the trace.

Nothing broke. What failed was the binding between the record on screen, the conversation state, the representative's authority, the proposal she confirmed and the write that committed.

What's wrong with a model wrapper?

Nothing, for experiments and read-only tasks. It becomes a problem when "screen, prompt, model, response" turns into the production design for work that changes records in other systems.

That path describes the most visible interaction, not the application. A production system also has to hold identity, tenancy, data eligibility, context sources, rules, durable state, human judgment, tool authority, side-effect handling, recovery, cost, latency, versions, monitoring and evidence. NIST's AI Risk Management Framework treats AI the same way: as a lifecycle and system concern involving many actors and contexts, not an isolated model.

Production machine-learning research reached that conclusion years ago: the model is often a small part of the real system, and data dependencies, glue code, configuration and monitoring create most of the operational risk. Generative models widen what the model can do. Tool use, dynamic context and long-running work make the surrounding obligations heavier, not lighter.

Systems of engagement on top of systems of record

Enterprises already have systems of record: CRM, ERP, core banking, HR, claims. They hold the authoritative facts and they're hard to change. AI applications are systems of engagement on top of them: the place where people and agents read, decide, approve and act.

That framing settles a lot. The system of record stays the source of truth. The AI application never reaches in and edits it freely; it proposes, checks, gets approval where needed and then makes a governed, confirmed change. The screen a person sees, the agent's reasoning and the write all have to agree on the same record, the same person and the same rules.

That boundary is where you can answer five questions: what outcome was intended, which capabilities and data were allowed to influence it, under whose authority it ran, which change actually happened, and what evidence lets it be operated, challenged, replayed and improved. A model contributes to each answer. It can't own all five.

What the model can do, and what it can't own

What must an AI application own that the model can't?

Ten things: purpose, identity and authority, context, decision meaning, policy and human judgment, durable state, safe actions, failure handling, cost and performance, and evidence with change control.

Not every application needs all ten at full strength. A drafting aid needs a light touch; an app that changes financial or customer records needs much more. But each obligation needs a named owner.

1. Purpose and boundary. Who it's for, what it's for, what it must never do, and the point where a model suggestion becomes a business decision. A model card describes intended use; the application enforces it.
2. Identity and authority. Who started the work, who's acting now, for which tenant, entity and purpose, with what delegated scope and expiry, carried through search, model, approval and tool calls. A service credential proves a service may call an API, not that this person may change this record.
3. Data and context. Which records, documents, features and prior decisions are eligible, current and permitted. Bad data cancels out model gains.
4. Decision meaning and contracts. The difference between a proposal, an observation, a verified fact, a policy decision, a human judgment and a committed action. A schema checks structure, not truth or permission.
5. Policy and human judgment. Rules, approvals and separation of duties enforced at runtime. A prompt can ask for compliance; it can't enforce it. Approval binds to the exact proposal and version that will execute.
6. Durable state. Surviving waits, retries, events, concurrent edits and record switches without changing what the case means. Conversation memory isn't application state.
7. Safe actions. Allowed tools, typed arguments, authority, operation IDs, reconciliation, reversal and evidence. A correct answer isn't a safe action.
8. Failure and recovery. Explicit outcomes for model outages, invalid output, missing context, policy denial, low confidence, approval timeouts and uncertain writes. "Try again" isn't a strategy.
9. Cost and performance. Settle what rules can settle, use a smaller model where it's enough, escalate uncertainty, and stop when the budget can't be met. (See deterministic-first routing.)
10. Evidence and change. Pinned versions, recorded dependencies, linked telemetry, decision records and context sources, and outcome monitoring.

The model enters as a governed step

Once the model is one capability rather than the center, its interface gets clear. A governed model step declares: its task and what its output means; typed inputs and outputs; which models, providers and regions are eligible; its prompt, tool and retrieval dependencies; its evaluation profile, limits and when it abstains; the identity, scope and data classes it requires; cost, latency, timeout and fallback; whether its result is advisory or needs verification; and what evidence it must leave.

That also makes providers swappable without pretending models are interchangeable. A replacement either meets the same declared contract and evaluation, or the application definition changes on purpose.

Evaluate the path, not just the model

Offline evaluation asks how well a model performs on a defined test set. Application evaluation asks how reliably the whole path produces the intended outcome under real conditions. Test the cases that cross team lines:

• a correct model answer paired with the wrong record or tenant;
• relevant evidence that's stale or outside the requester's permission;
• a valid proposal that breaks a business rule;
• an approval followed by a material change to the proposal or policy;
• a provider timeout after the tool already committed;
• a low-confidence result routed to review or abstention;
• a model route change that shifts cost, latency or tested coverage;
• a safe stop with no side effect.

These are hard to test when the application exists as prompts, adapter code and dashboards. They're straightforward when the workflow, contracts, policies and evidence form one inspectable definition. And evaluation continues after launch: outcome monitoring catches drift in data, context, approvals, tool failures and cost even when model metrics look stable.

Someone has to own the outcome

Component ownership isn't outcome ownership. The model team owns evaluation, data owns retrieval, security owns policy, the app team owns services, operations owns incidents, the business owns approval rules. Each can meet its own goals while the application loses meaning at the handoffs.

Someone has to own the whole outcome and the invariants that connect the parts: the record being changed, the authority being used, the proposal being approved, the state being resumed, the write being committed and the evidence kept. Without that owner, a model wrapper becomes an organizational wrapper too.

A prompt isn't a control plane

Prompts are good for task instructions, examples, tone and output shape. They're poor places for controls that need reliable enforcement and independent evidence. Don't let a prompt be the only place that defines which tenant may be accessed, which tools are available and under whose authority, whether an amount exceeds a limit, whether approval is mandatory, whether an action is reversible, how a paused case resumes, what evidence is kept or what happens when the model is unsure. Those belong in contracts, policy, runtime state and the checks before each write.

Tool use is where a feature becomes a risk

A read-only answer can be wrong. A tool-using application can change the world. Between the proposal and the consequence, the runtime should check the exact record and operation ID, the authority of who started it and who's acting, the approved proposal and any material changes, policy conditions, schema and domain validity, retry and reversal rules, and the confirmation from the system of record. The model may pick and fill a tool call. The system decides whether it may cross that line. IBM's 2025 breach research found 97% of organizations with an AI-related breach lacked proper AI access controls; this check is where those controls live.

The screen and the reviewer are part of this, too. The UI sets which record is active, shows evidence, captures confirmation and limits available actions. If the visible record and the hidden execution state can diverge, as in the opening story, the screen is part of the control problem. (See human-in-the-loop approval.)

Common anti-patterns

Ten rules for complete AI applications

1. Treat the model as one capability, not the application.
2. Build AI apps as systems of engagement on top of systems of record.
3. Bind every write to the exact record the person saw and confirmed.
4. Act with delegated authority, never a broad service credential.
5. Keep durable state out of conversation memory.
6. Enforce rules in policy, not prompts.
7. Check identity, record, policy and approval before every write.
8. Design failure states before adding autonomy.
9. Evaluate the whole path, not just the model.
10. Name one owner for the business outcome.

The result: judgment you can act on

A stronger model improves interpretation, extraction, ranking, drafting and planning. That matters. It isn't the same as an application you can run. Readiness belongs to the whole system, because the outcome depends on model quality, data, context, identity, policy, state, integration, human judgment, failure handling, cost and evidence together.

Keep the architecture proportional: a read-only drafting tool needs a light frame; an app that changes customer, financial or supplier records needs a strong one. The model is valuable for what it can infer. The application is trustworthy for what the system around it makes explicit, enforced, durable and explainable.