Token efficiency and green AI: cut the work, not just the prompt

The prompts got shorter. The case got heavier.

A bank runs an AI application for business onboarding: beneficial ownership, sanctions exposure, missing documents. The original route checks identities exactly, calculates ownership percentages in code, retrieves only the relevant filings, and calls a specialized model only when a legal relationship is genuinely ambiguous. Most simple cases finish with one bounded model call.

An efficiency push targets average input tokens per call. The prompt shrinks by 22%. At the same time, the application is rebuilt as a general multi-agent route. A planner receives the full customer dossier. An extractor, a verifier and a critic each get overlapping copies of the same material. A failed schema check repeats part of the route. Low-confidence cases go to a larger model with nearly the same context.

The token dashboard celebrates: every prompt is shorter. The whole case tells a different story. Total tokens per resolved case go up, because the same evidence is assembled and sent several times. Retrieval and reranking repeat. More cases reach human review because the agents' summaries disagree, and reviewers reopen the source documents to untangle contradictions the route created.

Are tokens a good measure of AI efficiency?

They're a useful signal, not a measure of efficiency. Tokens count what crosses the model boundary; they don't capture hardware, utilization, energy, carbon, retries, human rework or whether the result was right.

The same token count can mean very different work depending on model architecture, hardware, batch size, precision, serving stack and utilization. A model call with fewer tokens can create more downstream work if it drops evidence, fails validation or triggers human correction. So keep several measures, and don't let any one stand in for the rest:

Start with a boundary and a unit of work

Green AI research argues that computational cost should be reported alongside model quality, not treated as invisible. NIST's generative AI profile names environmental impact from heavy compute use as a risk to manage across the lifecycle. The Software Carbon Intensity specification starts the same way: declare the software boundary, then declare the functional unit you measure against.

For an enterprise application, the model endpoint is rarely the whole boundary. Retrieval, feature calculation, deterministic code, durable state, policy, human review, retries and integrations all belong inside it. The right functional unit is the business result: one resolved claim, one completed onboarding, one reconciled payment. That's the same unit you use for AI cost attribution, and it stops you reporting great efficiency per call while ignoring the extra calls and correction it took to reach a valid outcome.

Where do the biggest token savings come from?

From work you never do. Skipping an unnecessary model call removes its prompt, output, queueing, compute, provider dependency and failure surface all at once, which beats any amount of prompt trimming.

There are two ways to improve efficiency: do necessary work more cheaply, or prove some work was never necessary. The second usually wins.

Avoided inference still has to be governed. Skip a model only when the alternative meets the same quality, evidence and policy bar. The goal isn't the fewest model calls; it's the least work consistent with what the decision requires. (See deterministic-first routing.)

Context is usually the biggest avoidable input

Prompts grow because every component gets a convenient copy of the whole case. A planner, classifier, verifier, summarizer and reviewer model may each process the same instructions and documents. You pay to send and reason over that material again and again.

Blunt prompt compression isn't the fix. Removing decisive evidence lowers input tokens and raises wrong answers, escalations and rework. Instead, sort context into four tiers:

• mandatory evidence that survives every budget decision;
• conditional evidence triggered by the current state or policy;
• optional evidence that helps but can be dropped;
• prohibited evidence that must never reach the model.

Then pass verified facts (identifiers, calculated amounts, statuses) as typed values instead of making each model rediscover them from documents or from an earlier model's summary. Token reduction becomes a result of better application design, not the goal you chase directly.

Output tokens are a design choice

If a step needs a three-field classification, don't ask for a page of narrative and parse it back into three fields. Ask each consumer for the smallest output it needs: typed fields for the policy check, a short rationale for the reviewer, only the authorized instruction for the system that acts. Smaller, typed outputs cut generated text, parsing errors and storage, and they reduce the chance that a later step treats a model's explanation as an instruction. And define stopping behavior: a model that has met its output contract shouldn't keep generating because the token limit is generous.

When does caching actually save work?

When the reused result is still valid for this case: same tenant and scope, same purpose, same source and model versions, same policy, and still fresh. Otherwise a cache hit saves one call and creates a wrong decision.

A cache keyed only on prompt text or a resource ID is usually too weak. What changes behavior includes the tenant, data scope, purpose, policy revision, source revision, model version and effective time. Reuse applies to prompt segments, response caches, retrieval results, verified facts and features, and deterministic calculations, each under explicit validity rules.

Measure reuse by work avoided and outcomes produced, not hit rate. A high hit rate can hide stale answers and cross-scope leaks.

Right-size the model, not just the prompt

Once exact and redundant work is gone, pick the smallest allowed capability that meets what's left:

• deterministic code for exact questions;
• a narrow classifier or extractor for a bounded task;
• a specialized model where its evaluations cover the task;
• a general reasoning model only for genuinely open-ended ambiguity;
• a person where accountable judgment is required;
• batched or scheduled runs for work that can wait.

No model is always the cheapest, fastest or lowest-energy choice. Inference energy depends on input and output length, batch size, hardware, precision, serving stack and utilization. A stronger model can be the efficient pick when it prevents several failed attempts and an escalation; a smaller one wins when it meets the same requirement directly. Research on model cascades shows routing by task can improve cost-performance, but in enterprise settings the cascade has to stay inside policy. (See model routing across cost, latency, quality and policy.)

Escalation should earn its cost

Every extra retrieval, verification, model call or review step should reduce a specific uncertainty. A self-critique loop that writes longer prose but doesn't change validation, evidence or the decision adds cost without value. A targeted fetch of one missing amendment adds a lot. A reviewer who resolves a policy-defined ambiguity justifies real time, because no model can close that question with authority.

Retries and rework multiply hidden waste

Much of the waste happens after the first attempt:

• a model returns invalid structure and the whole context is resent;
• a missing identifier breaks a tool call and the agent replans from scratch;
• a timeout triggers another write before anyone checks what happened;
• a thin review package sends the reviewer back into source systems;
• a weak evidence trail creates an audit investigation months later;
• a wrong effect needs a reversal and customer remediation.

Typed failures, a stable ID for each business action, reconciliation before retry, and complete review packages are green controls as much as governance controls. The greenest run isn't the one with the fewest tokens before the first answer. It's the one that reaches a valid outcome with the least total work, including the work to explain, correct or recover.

Energy and carbon aren't the same thing

Energy asks how much electricity the software uses per unit of work. Carbon also depends on how that electricity was generated, and on how embodied hardware emissions are allocated. The same energy can mean different carbon in different places and hours.

That matters for routing. A deferrable batch job may run when lower-carbon capacity is available. An interactive or regulated decision may not be deferrable, and data residency, provider approval, latency and quality constrain the route before carbon ever enters the decision. Private AI isn't automatically greener for being private, and shared public inference isn't automatically more efficient for being shared. Utilization, hardware, model size, workload shape, cooling, electricity source and data movement all count. Keep tokens, compute, energy and carbon as separate measures and compare them per outcome.

Measure efficiency per outcome

Useful measures, none of which should become the sole target:

A route can cut tokens while adding human work, cut energy while breaking residency, or raise cache hits while serving stale answers. Efficiency always sits inside quality, authority, policy and evidence constraints.

Common anti-patterns

Ten rules for token-efficient, green AI

1. Declare the boundary and the outcome unit before optimizing tokens.
2. Treat tokens as one signal, not a proxy for energy, cost or quality.
3. Settle exact questions without a model, and record that you did.
4. Send each step only the context it needs; never drop mandatory evidence.
5. Ask for the smallest typed output each consumer needs.
6. Reuse context and results only while scope, version and freshness hold.
7. Right-size models within the allowed set; don't default to the largest.
8. Make every escalation earn its cost, and stop when requirements are met.
9. Count retries, rework, human correction and recovery in the total.
10. Keep the link from resource use to outcome, so you can show what was necessary.

The result: less work per outcome

Shorter prompts, smaller models, quantization, batching, caching and carbon-aware scheduling are all useful. None of them proves the application is efficient. The test is whether the whole route does the least work needed to produce a valid, authorized, evidence-complete result.

That usually means eliminating work before optimizing it: exact computation before inference, required context instead of everything, typed facts instead of repeated prose, targeted escalation instead of reflexively bigger models, and reconciliation before retry. Efficiency isn't fewer tokens per call. It's less unnecessary work per outcome.