Retries and idempotency for AI agents: how a timeout becomes a double payment

The payment timed out. The money didn't wait.

An agent reviews an insurance claim (the file, medical evidence, policy terms and adjuster notes) and proposes a $96,000 settlement. Checks confirm the policy limit and currency. Policy requires senior approval, and an authorized adjuster approves the exact proposal.

The workflow calls the payment service. The service commits the transfer and creates a receipt. Then the network drops before the response arrives. To the caller, the request timed out. To the payment service, the money has moved.

A conventional retry policy kicks in. It builds a new request with a new idempotency key. The payment service sees a new instruction and pays another $96,000. Reconciliation finds two payments days later.

Every part behaved as designed. The failure was treating "no response received" as "no effect occurred."

Is a timeout a failure?

No. A timeout is evidence of uncertainty, not of non-execution. The request may never have left, may have been rejected, may have failed midway, may have committed and lost its response, or may still be running. One symptom covers several very different business states.

HTTP already draws this line: clients shouldn't automatically retry non-idempotent requests unless they know the operation is safe to repeat or can show the original wasn't applied. Agent workflows need the same discipline at the business level, where one action spans proposals, approvals, records and long waits.

Five layers of one action

Retry bugs are easier to reason about once you separate five layers that usually get collapsed into "an API call":

One operation ID, kept across every retry

An attempt ID answers which try is this? An operation ID answers which single business intention are all these tries for? Create it before the first attempt, and bind it to what was authorized: the tenant, record and action; the proposal and its fingerprint; the amount and beneficiary; the authority, policy decision and approval.

Each retry gets a new attempt ID. The operation ID never changes, because nobody authorized a second payment.

A payload hash isn't a substitute. Two legitimate payments can have the same amount and beneficiary for different invoices. A retry can differ in a timestamp and still be the same operation. The ID has to come from business meaning, not byte equality.

Six mechanisms that aren't synonyms

Event standards such as CloudEvents let a re-sent event keep its ID so consumers can de-duplicate it. That helps at the event level. It doesn't give a payment or contract the stable business identity it needs across many events and attempts.

Declare how each action behaves under failure

A schema describes what a payment request looks like. It doesn't say what to do when the response disappears after the payment commits. Each action that changes something outside your system needs a declared side-effect contract:

Idempotency keys: four rules

1. Reuse the same key for every retry of the same action.
2. Never reuse a key for a materially different action: a new amount, record, authority or payload.
3. Know the target's retention window. If a workflow can resume after the target forgets the key, you need another recovery path.
4. Coordinate concurrent attempts. Two workers with the same key can still race if the target doesn't define concurrent behavior.

Record the request fingerprint alongside the operation ID, and reject a different request under the same ID. Don't generate a new ID just because a different worker, model or provider picked up the work.

"Unknown" is a real state

A timeout after a write shouldn't jump straight to failed. The honest state is uncertain, and it stops blind retries. The lifecycle runs from proposed to authorized to in flight, and then ends in one of three outcomes that aren't interchangeable:

• Committed: a receipt or reconciled state proves the effect.
• Failed before effect: you can show nothing was applied, so a retry may proceed.
• Uncertain: you can't yet prove either, so don't invent certainty.

What should an agent do after an uncertain outcome?

Reconcile before retrying: ask the target system what happened to that operation ID, then recover the receipt, retry with the same ID, or escalate. When an outcome is uncertain, find out what happened before doing anything else. Query the target by operation ID or idempotency key, match a later webhook, check the ledger or record version. Then answer four questions:

1. Did any effect occur for this operation?
2. If so, does it match the authorized proposal, amount, record and beneficiary?
3. Is the state you found authoritative and complete?
4. What may happen next: recover the receipt, retry with the same ID, stop, escalate or compensate?

If the target confirms the original effect, recover the receipt; don't retry. If it proves nothing happened, retry with the same operation ID. If it can't say, send it to a person.

A reversal is a new action, not a rollback

Once an external effect is committed, you can't rewind the world. You can take another action: refund, revoke access, send a correction, post a reversing entry, cancel a shipment. That action has consequences of its own and can fail too. So it needs its own operation ID, a fresh authority and policy check, human approval where the stakes call for it, a link to the original operation, and its own record. Calling it "rollback" hides the fact that you've done something new.

Classify before you choose a retry policy

One global retry policy is unsafe because actions carry different consequences:

The class isn't purely technical. The same API call can be low-stakes in a sandbox and high-stakes in production, depending on amount, sensitivity and reversibility.

Agents make retry ambiguity worse

After a timeout, an agent focused on finishing its goal may try again, switch provider, call a similar tool, replan through another path, reword the request or mint a new ID. Each choice can look reasonable for the goal while breaking the rules for the action. OWASP's agentic risks include exactly this kind of tool misuse, including runaway volume.

So the agent shouldn't own the operation ID. Its tool request arrives as a proposal. The runtime assigns or validates the ID, checks policy and approval, applies the side-effect contract, and decides whether to execute, reconcile or escalate.

Retries must keep identity, policy and approval intact

• Events: a duplicate event may trigger another attempt; it must not create another operation.
• Sub-workflows: parent and child must agree on who owns the ID, retries and confirmation.
• Versions: a paused operation stays bound to the version and proposal it started under.
• Identity: every retry keeps or rechecks the authority it runs under. (See AI agent identity.)
• Approval: a retry of the same operation can rely on the approval; a changed one can't. (See human-in-the-loop approval.)

Common anti-patterns

Ten rules for safe retries

1. Classify the effect before choosing retry behavior.
2. Create one operation ID before the first attempt.
3. Keep it across every retry, worker, queue, sub-workflow, provider, wait and resume.
4. Bind it to the exact proposal, authority, policy, approval and terms.
5. Treat a timeout as uncertain unless you can prove nothing happened.
6. Reconcile before retrying an uncertain action.
7. Use the target's idempotency support with known scope, retention and concurrency.
8. Keep retry, de-duplication, reconciliation, compensation and replay separate.
9. Record each action through to the target system's confirmation.
10. Let agents propose recovery; let the runtime decide.

The result: resilience without duplicated reality

Retries aren't the enemy. They're how distributed systems survive partial failure. The mistake is letting a technical retry silently become a second business intention. One stable operation ID, an honest "uncertain" state, reconciliation before retry and governed reversals don't promise exactly-once behavior across arbitrary systems. They give you something more defensible: one declared intention, controlled attempts, explicit uncertainty and proof of what actually happened.