A cache hit removes 93% of an agent's prefill energy. The other 86% is untouched.

Prefix caching is sold as the fix for long-context agents. It removes the cheap half.

A prefix cache hit removes 93% of an agent request's prefill energy. What it cannot touch is the other 86%: the 68,000-token KV cache that every generated token has to read again.

That sentence is the whole paper. Everything below is just the arithmetic and where the number came from.

The number everybody will quote, and the one they will skip

Anthropic-style prefix caching is sold as a discount, and the discount is real: on a long-context agent request, prefill energy falls from 1,207.8 J to 88.8 J, a 92.6% cut. If you stop there, you conclude that caching fixes agents.

Stop one column later. The same table, Table 8 of the SIMPLE paper, prices three things per request: prefill, KV-cache re-read, and the next token. With the cache hit applied, only the first row moves.

A fully cached agent request still costs 4.84× a chat request, and its marginal output token — the cost of producing one more word — stays at 2.35 J against 0.20 J, an 11.75× gap that caching does not move at all.

The reason is mechanical rather than economic. "Cache hit" means the system does not re-compute the prompt. It does not mean the model stops re-reading it. Every decode step attends over the whole KV cache, so with a 68K context, 247 output tokens means 247 passes over 68K of cached state. That re-read is 573.8 J, and it is 86% of what is left after the cache has done its work.

What the law actually is

SIMPLE is a four-parameter closed-form energy model, and its structure is the reason the table above is computable at all. It splits a request into prefill and decode, and splits decode again into a KV-cache term and a next-token term. Both MHA and GQA get closed-form coefficients.

Two claims matter for anyone outside the paper.

First, it generalises: 13 dense models, six families, one H100, R² = 0.969 and MAPE 11.97%, with a single fit and no per-model calibration, which the authors report as 3.1–3.5× lower error than the SweetSpot baseline. You can therefore price a 68K-context workload with a model that was fitted on sequences of at most 4,096 tokens.

Second, it explains rather than merely fits. Prefill energy grows quadratically with hidden dimension and linearly with layer count; more aggressive GQA grouping lowers per-token energy by up to 3×. Those are architecture knobs, not billing knobs — a distinction that matters in the last section.

When a model is fitted on 4K and used at 68K, say so. The paper's Table 8 caption calls its agent row an estimation, and its limitations list MoE, batching and multi-GPU but not the 16× length extrapolation. Our reading: trust the direction, distrust the decimals. Writing "about 5×" is honest; writing "4.84× measured" is not.

Where the 68K comes from

The workload side is not this paper's experiment. It is quoted from a production trace study (arXiv:2608.00101): Copilot traffic with a median context of 68K tokens, median completion 247 tokens, and a 63K cached prefix — roughly a 92.6% hit rate — while ordinary chat prompts in the same population run 750–1,050 tokens.

That is the important structural fact. The fastest-growing class of requests is exactly the class that sits in the cell where the cache is already saturated and the KV term dominates anyway. The cache is not failing; it is working perfectly on the part that is no longer the bulk of the bill.

Three details that change how you read it

1. The 4.8× is an extrapolation. Fitting sequences stop at 4,096 (max 8,192); the agent row is computed at 68K. Nobody measured 68K here.

2. The KV term scales with input and output. It is proportional to context length per generated token, which is why the same word costs 0.20 J in a 1K-context chat and 2.35 J in a 68K-context agent loop. Token-metered pricing is blind to both multipliers.

3. Batching is not modelled. Real agent serving is highly concurrent; the authors say so. Directionally the conclusion holds — the KV re-read is independent of the cache hit — but absolute joules will move with batch size.

So what

If you price inference: a cache discount discounts prefill. Output tokens are priced flat regardless of context length, which makes long-context agents the one request class where the sticker price and the cost structure point in opposite directions. Tiering output by context length is the correction; discounting input alone is not.

If you build agents: cutting context (truncation, hierarchical memory, summarisation) beats switching on caching, because caching removes the cheap half. The other real lever is attention structure — GQA grouping cuts the KV term by up to 3× — and that is an architecture decision, not a config flag.

If you buy capacity: tokens-per-joule mixes two request classes that differ by 13× in energy. Reporting one blended number hides a structural gap and will not survive the first agent-heavy workload.

What this piece does not claim

The paper's intelligence-per-joule section uses third-party AIME-24 accuracy, so its "capability per joule" conversion depends on someone else's benchmark. We use only the energy rows. Likewise, the 2.35 J marginal-token figure is defined as the cost of one more generated token with prefill already paid, which is precisely the right comparison against 0.20 J — but it is not a claim that an agent request costs 11.75× a chat request end to end. End to end, it is 4.84× with the cache, 12.94× without.

One objection is fair: real serving stacks use chunked prefill and high concurrency, so the split inside a request shifts. The conclusion survives it, because "the KV re-read does not care whether the prefix was cached" is an analytic statement about attention, not a benchmark artefact.

The cache is not the problem. Believing that a cache hit is a cost reduction is.

How this could be wrong

Three ways, and each is testable without rerunning the paper.

First, if the cache-hit row in a real stack also removes the KV re-read — for instance because a serving system keeps the whole KV resident and prices it as a fixed allocation rather than a per-step cost — then the 86% is an artefact of how energy is attributed, not of how it is spent. The paper's model charges KV reads per decode step; a provider with unlimited fast memory might amortise them differently, but it cannot make the reads disappear.

Second, if agent frameworks stop being long-context. The 68K median comes from a Copilot trace where the repository and conversation are in the prompt. If the next generation of tools moves state into retrieval and keeps the window near 4K, the whole cell empties and this argument loses its object. That is a plausible outcome, and it would be the good ending.

Third, if context length stops being the dominant energy variable at serving scale because batching dominates. High concurrency amortises weight reads across requests; it does not amortise the KV term per request, so the ratio narrows but does not invert. Any published per-request energy breakdown with batch size attached would settle it — the paper's own limitations list batching as unmodelled.

What would move us: a provider publishing joules per request by context bucket, with batch size and cache state. Until then, Table 8 is the best available arithmetic, and it says the cheap-looking optimisation is aimed at the wrong row.

The practical version is boring. If you are paying for long-context agents, the item that grew is re-reading state, not computing it — so the fix is smaller state, not a larger discount.