AI behavior explainability: inspect what can, does and did happen

Every dashboard was green. Six months later, nobody could say why.

An AI-assisted underwriting application raises a business customer's credit limit. The request completes in under three seconds. Retrieval succeeds, the model call returns cleanly, the policy service responds, and the account API accepts the update. Every dashboard is green.

Six months later the decision is challenged. The investigator doesn't ask whether the services were healthy. They ask which documents the application was allowed to retrieve, which passages reached the model, which financial figures were current at that moment, which model and prompt versions ran, which policy facts were checked, whether anyone overrode the result, and what authority the application had when it changed the account.

The team has traces, logs and metrics. What it doesn't have is a way to connect them to what the application was supposed to do, what it actually knew, and why the path it took was allowed.

Observability is doing its job

Metrics summarize behavior over time. Logs record events. Traces follow a request across services and show timing and parent-child relationships. Trace context standards carry the identifiers that let you correlate one transaction across systems. Audit records show that an event happened, when, where, with what outcome and under which identity.

Those answer indispensable questions: Is the service up? Where is latency building? Which dependency failed? Did a retry happen? Which model endpoint was called, and how many tokens did it use? None of that should be discarded.

OpenTelemetry's generative AI conventions are extending this vocabulary to models, agents, tools and prompts, though much of it is still marked as in development and prompt and output content is opt-in, sensibly, given privacy and volume. Richer telemetry is useful. It still records what instrumentation saw. Pipelines sample, redact, aggregate and expire data by design, and even a complete trace doesn't say which policy version was authoritative or whether context respected the user's access.

The point isn't observability versus evidence. It's observability plus evidence. (For the record-keeping side, see AI agent audit trails.)

What does it mean to explain AI behavior?

Explaining AI behavior means being able to show, from structured evidence, what the system was allowed to do, what it was doing at any moment, and how a specific outcome was reached. That's three different moments of inspection, and each needs its own support.

Most teams invest only in the third moment, and only through logs. The first two are where most investigations could be avoided.

Design time: can you see what's allowed to happen?

At design time, anyone responsible for the system should be able to read which paths exist, where models are used, which rules and approvals guard each action, and which versions are in force, before anything runs.

That only works if the thing you read is the thing that runs. When logic is spread across prompts, model adapters, config files, policy consoles and runbooks, there's no single place to answer "can this agent change a credit limit without a person?". A readable workflow definition, executed as written, makes that a question you answer by looking rather than by testing in production.

Design-time inspection also covers change. Before a new model, prompt or policy goes live, you want to see every workflow that depends on it, compare the new version with the old, and run it against recorded cases. That's where you catch a rule that sends every low-confidence case to a bigger model, or an action step with no approval before it.

Run time: can you see what's happening now?

At run time, you should be able to see where each live case is, which step it's on, what it's waiting for and why, and which authority and policy apply to its next action.

Agent work isn't always a three-second request. A case can pause for a document, an approval or a timer, then resume days later. During that time, operations needs to know which cases are stuck, a reviewer needs the whole case rather than a notification, and security needs to know the agent is still acting under the right person's authority. A green service dashboard can't show any of that, because no service is failing. The case is simply waiting.

Run-time visibility also means telling apart things that look alike: a proposed tool call versus the validated one, a request sent versus an effect confirmed, a timeout versus a failure. "The tool was called" and "the account changed" aren't the same fact.

A timeline is not an explanation

A trace can show that retrieval ran, then a calculation, then the model, then policy, then approval, then the update. That's a path. It doesn't show which of 40 candidate documents were allowed and selected, which values crossed a threshold, which model output fields were accepted or repaired, which facts the policy decided on, or whether the approval covered the version that ran.

Order in time isn't influence. A document can be retrieved and excluded. A value can be present and irrelevant to the rule. A model can propose something a deterministic check replaced. NIST's work on explainable AI separates the correctness of a decision from the accuracy of its explanation: an explanation should reflect what the system actually did, not just tell a convincing story.

Structured evidence shows the inputs, versions, constraints and actions visible at the application level. It shouldn't claim to expose the model's internal reasoning. The aim is defensible application cause and effect.

After the run: can you show how the outcome was reached?

After the run, you should be able to reconstruct a specific outcome from evidence: the definition and versions in force, the authority used, the context allowed in, the decision factors, the actions taken and the confirmed result.

For consequential actions, that record has six parts:

Without version binding, an investigation quietly substitutes today's system for yesterday's behavior. Completeness has to be explicit too: a record that says "this item was redacted" or "this dependency couldn't be resolved" is more trustworthy than a polished view that silently omits the gap.

Size the record to risk. A low-stakes summarizer needs much less than a system that changes credit limits or releases payments. Don't copy every sensitive value into a second store; use references, digests and protected snapshots. (The decision part is covered in decision lineage.)

One record, different views

A developer needs payloads, schema errors and run comparisons. An SRE needs latency, retries and cost by step. Security needs identity, data scope and policy decisions. An auditor needs versions, approvals, origin and integrity. A business owner needs the decision factors, exceptions and resulting state.

They shouldn't investigate five different stories. Give each role a controlled view over the same record. Access control can change how much someone sees, but it shouldn't create incompatible histories of what the application did.

Replay is investigation, not retry

The strongest test of an evidence design is whether it can rebuild a past run without swapping in today's facts. Keep replay modes distinct:

• Inspect: read the past definition, evidence, decisions and actions. Nothing executes.
• Reconstruct: resolve the stored versions and snapshots needed to rebuild the past state.
• Simulate: run the past or an alternative version in a sandbox, with tools stubbed and side effects suppressed. "What would today's policy have done?" is a simulation, not history.
• Re-execute: a deliberate new run with fresh authorization and its own record, linked to the original.

A retry is different: it tries to complete an intended action after failure. HTTP semantics already warn against automatically retrying non-idempotent requests. Confuse replay with retry and you duplicate payments, emails and account changes. And be honest about limits: a hosted model may change behind an alias, and a live index may no longer hold the same content. Surface those gaps rather than invent certainty.

Common anti-patterns

Ten rules for explainable AI behavior

1. Make the definition you inspect the one the runtime executes.
2. Show every action's guards (policy, approval, authority) before anything runs.
3. Review dependencies and compare versions before a change goes live.
4. Show each live case's current step, wait reason and authority.
5. Keep proposed, validated and confirmed actions distinct.
6. Bind every consequential run to the exact versions it used.
7. Record decision factors as structured evidence, not prose.
8. Mark evidence gaps explicitly.
9. Give each role a view over the same record.
10. Keep inspect, simulate, re-execute and retry separate.

The result: a system you can see into

Telemetry tells you the machine is running. Explainability tells you what the machine is allowed to do, what it's doing now and how it reached a particular result. Teams that build for all three moments catch design flaws before launch, spot stuck or misdirected work while it's live, and answer a challenge six months later with a record instead of a reconstruction exercise.

Visibility alone isn't governance. But you can't govern what you can't see, and you can't defend what you can't show.