Confidence escalation for AI agents: route uncertainty, not just scores

The bigger model was more sure. The exclusion was still missing.

An insurer uses an AI agent to triage property claims. A homeowner reports water damage after a pipe burst. The base policy covers sudden water damage. An endorsement added at the last renewal excludes damage to properties left unoccupied for more than 60 days, and this property had been empty for three months. The endorsement sits in the policy system, attached to the right policy. A delayed index update keeps it out of the documents retrieved for this claim.

The first model reads the base policy, answers "covered" and flags low confidence. The routing rule reads low confidence as "not smart enough" and sends the same incomplete context to a larger model. The larger model writes a cleaner analysis, cites the coverage clause and reports high confidence. The claim clears the auto-approval threshold and $48,000 is paid.

Every component did its job. Retrieval returned what was indexed. The first model honestly signaled doubt. The second reasoned well over what it was given. The output passed its schema. The failure was escalating model capacity before anyone asked why the first answer was uncertain.

Is a confidence score a measure of risk?

No. A confidence score describes one model's view of its own output, not the business risk of acting on it, and different kinds of score answer different questions. Treating them as one number, and that number as permission, is where most escalation designs go wrong.

A model can be very sure of its reading while the application stays uncertain, because a required document never arrived. Two models can agree because they share training data or the same thin context. A check can confirm the arithmetic and leave authorization open. Research on modern neural networks has shown they're often miscalibrated: a stated 90% doesn't mean right nine times in ten.

Low confidence is a symptom. Diagnose it first.

An application can be uncertain for very different reasons, and each needs a different fix.

The goal isn't the highest score. It's a valid, authorized, evidence-complete outcome, or an honest stop.

Fix the evidence before you change the model

If a mandatory endorsement, current balance or policy obligation is missing, you have a context problem. A stronger model reasons better over what it receives. It can't recover what it was never shown. That's why the context check comes first: is everything the decision requires present, current and permitted? Only when it is should leftover uncertainty be treated as a reasoning problem. (See context assembly beyond RAG.)

Keep the outcomes typed: context incomplete, context stale, access denied, sources conflict. Collapsing them into one low score throws away the very information that tells you what to do next.

Exact checks are often the best escalation

Many uncertain outputs contain parts you can verify exactly. Compare an extracted value to its source passage. Normalize a date against the policy's time zone and calendar. Recompute an amount from typed values. Resolve an identifier against the system of record. Check a proposed change against the record's current version.

A check can confirm, repair within declared rules, reject, narrow, or send the result back for more evidence. It shouldn't quietly turn an unsupported claim into a fact. The model contributes interpretation; deterministic steps hold the exact obligations. Every part you verify is ambiguity a later model or reviewer doesn't have to carry.

When is a stronger model the right escalation?

A stronger or specialized model fits when the evidence is complete, the exact checks pass, and what's left is genuine reasoning or interpretation difficulty. Ask which capability can reduce this uncertainty, not which model is biggest.

A specialized model may have narrower range but better tested coverage for the task. A larger general model may reason more broadly but cost more, run slower, or fall outside residency and privacy limits. Agreement among several models is useful but isn't independent verification; they can share sources, prompts and failure modes. Sampling the same incomplete context again measures instability, not adequacy.

When you do escalate to a stronger model, pass forward what's already settled: verified facts, open conflicts, source references, policy limits and the reason for escalating. Don't make it rebuild the case from prose.

When should a person review an uncertain AI decision?

A person should review when explicit rules say the case needs judgment: high consequence, contested facts, a policy that requires sign-off, or a case outside tested scope. A model's confidence alone shouldn't decide it.

Routing every low-confidence case to a queue looks safe and isn't. It creates delay, inconsistent decisions and reviewers who start clicking approve. Gartner analysts have warned about exactly this approval fatigue in one-size-fits-all agent governance. When volume floods the queue, the cases that genuinely need judgment get the same thin attention as the rest.

When a case does reach someone, give them the whole case: what's still uncertain, what earlier steps established, the alternatives, the consequence, and the exact action they're authorizing. The reviewer needs the expertise and the authority, not just a login. Their decision should be recorded as a durable approval tied to the exact proposal, not a note that "a human was in the loop". (See human-in-the-loop approval.)

Research on learning to defer treats people and models as complementary: a model may beat a reviewer on one kind of case and lose on another. Evaluate the combined route, not either side alone.

Stopping is a valid outcome

Selective prediction research formalizes the value of a reject option: accept fewer cases, make fewer errors on the ones you accept. Enterprise applications need the same move. If no available step can resolve the uncertainty within safety, policy, cost or time limits, the system should stop without acting and record why.

Typical reasons: insufficient evidence, outside tested scope, conflict needs review, no qualified reviewer available, escalation budget spent. Stopping doesn't have to mean an opaque error. The application can ask for information, explain what's unresolved, open an exception case or schedule another try. What matters is that uncertainty never turns into an unauthorized action.

Escalation needs a budget

An application that keeps retrieving, resampling, critiquing, calling bigger models or requesting review can burn time and money while looking diligent. Set limits per decision: context expansions, model calls, checks, elapsed time, cost and reviewer effort. Scale them to consequence. A low-risk draft can afford several cheap attempts. A time-sensitive payment needs a tight deadline, a fresh read of current state and a hard stop path.

Record why each step was tried and whether it reduced the uncertainty. Repeating the same step with the same context isn't escalation. Track a few numbers by route and segment: how often each step resolves the problem, how often confidence rose without better evidence, review volume and override rate, and the full cost to a governed outcome rather than one model call's price. As cases move from review to automation, the same records show where that's earned. (See progressive automation.)

Common anti-patterns

Ten rules for confidence escalation

1. Start with the decision and its consequence, not a threshold.
2. Diagnose what's unresolved before choosing the next step.
3. State exactly what the next step has to resolve.
4. Prefer more evidence, a fresh read or an exact check when they fit.
5. Tie every confidence threshold to a specific task, population and version.
6. Pass verified facts and open conflicts forward when you escalate to a model.
7. Let explicit rules, not model confidence alone, decide which cases reach a person.
8. Give reviewers the whole case and the exact action they're authorizing.
9. Treat stopping as a valid, recorded outcome with no business effect.
10. Set budgets and record why each step was taken and what it resolved.

The result: less uncertainty, not more confidence

Missing evidence should route to evidence. Stale state should route to the system of record. Exact claims should route to exact checks. Real ambiguity may justify a stronger model. High-stakes judgment, selected by rules, goes to a qualified person. Cases with no safe route stop without acting.

Designed this way, people review fewer cases and see the ones that need them. Models are used where they add something. And when someone asks why the system acted, or didn't, the answer is a record of what was uncertain and what resolved it, not a confidence number that sounded convincing at the time.