A latent-compression paper reports 89% fewer tokens and up to 56% less runtime. Its own appendix puts the compressor on a second A100. Multiply the two and the saving is 12% at best — and negative in five of eight settings.
96.19 s. That is the uncompressed agent in Table 1 of the paper, and 64.64 s is the compressed one. The same appendix says the compressor runs on an additional GPU. Charge both cards to the same account and the "faster" system is 96.19 GPU-seconds against 129.28 — 34% more expensive.
This is not a story about a bad paper. The method works: fewer tokens enter the context, answers come back sooner. It is a story about an accounting unit that quietly changed halfway through the evaluation, and about what happens to an efficiency claim when you multiply it back out.
Coca-Cola compresses every tool return into 128 continuous latent tokens (§A.2) instead of the 4,096-token text it replaced. A small language model does the compressing, conditioned on the task, the current reasoning step and the raw observation (§3.3), and it is trained by action-preserving distillation so that the next action distribution matches the uncompressed run — only a LoRA and an adapter are trained (§3.4, with 4,188 tasks, 4,363 observation–action pairs and 3,927 training pairs per A.1).
The results are real. Across eight settings token counts fall 70.01–89.24% and wall clock falls 24.45–55.97% (Tables 1 and 2), and exact-match or F1 improves in 13 of 16 cells — the GLM group wins all four, with TriviaQA F1 going from 44.57 to 59.41.
Then comes the sentence everything else hangs on (§A.3): "the target agent running on a single GPU in all settings. Where an additional GPU is used, it hosts only the compression model." The evaluation measures the agent's latency, on the agent's card. The compressor's work is on the other card, outside the stopwatch.
If the runtime halved but it now takes twice the hardware, GPU-seconds did not halve — they went 2 × (1 − r) where r is the reported reduction. For a 40% reduction that is a 20% increase. Run the paper's own eight rows through it:
Five of the eight rows get more expensive: the four Qwen settings by 34.4%, 45.4%, 49.8% and 51.1%, and GLM on HotpotQA by 16.8%. The three that still save — 5.5%, 8.1% and 11.9% — are all GLM. On average the eight settings cost 21.5% more GPU-seconds, and the kindest reading of the hardware note (assume the compressor shares the agent's card and the wall clock already includes it) merely returns you to the 24–56% saving the paper claims. There is no reading in which the token cut is a compute cut.
There is a physical reason the split is uneven. The agent is a 35B mixture-of-experts model activating 3B parameters; the compressor is a 4B dense model. Per token, the small dense model is not cheap enough to be free — so the pair saves 5–12% on GLM and costs 34–51% on Qwen. Which card the compressor lives on decides the sign of the result.
A second GPU is the visible half of the accounting gap. The invisible half is inside the code. In train/eval_all.py, compressor_tokenizer(..., max_length=4096) feeds _compressor_to_latent_hidden, while result["prompt_tokens"] only accumulates the agent-side prompt_len. Up to 4,096 tokens per call pass through the compressor and never touch a token column. The config agrees: configs/latte.example.yaml pins agent_num_gpus: 1 and compressor_num_gpus: 1.
The repository even writes the rule down. docs/METHODS.md, in its LLMLingua section, says the compressor's tokens and time are recorded in the compression field of the retrieval row and "should not be ignored when computing total computational cost." The same document, applied to this method, would have produced a GPU-seconds column. It was not applied.
None of this looks like intent. It looks like two teams of measurements meeting in one abstract: one that reports model tokens, one that reports wall clock, and a reader invited to multiply them.
Three honest objections, because the accounting gap cuts both ways.
Retrieval time may be inside the clock. If time_seconds includes the Exa round trip, then a constant is being added to both arms and the relative reduction is flattered. Code distinguishes generation_time_seconds, but the tables do not say which one they report. Budgeting 1 s per call over at most five calls, Qwen/NQ moves from 91.2 to 119.3 GPU-seconds (+31%) and GLM/NQ from 84.3 to 37.2 (still cheaper). The sign does not flip for any row.
The GLM group may not use a second card at all. Appendix A.3 says "where an additional GPU is used" — per-setting, not per-paper. If GLM co-locates the compressor, its three savings shrink to between 0% and 12%, and the honest headline becomes "+21.5% on average, five of eight worse", not "all of it is fake".
Latency is not nothing. An interactive agent that waits 64 s instead of 96 s is a better product, and paying twice the hardware for a third less waiting can be a good trade. The claim I am objecting to is narrower: that this trade is a reduction in computational cost. §5.1 says "practical improvements in inference latency and computational cost" — the abstract itself is careful enough to say only "reduces the agent's token consumption".
If you build agents and pay the GPU bill, your efficiency table needs two columns it probably does not have: GPU-seconds per task, and tokens through every model in the pipeline, not just the one holding the conversation. A compression layer that lives on another card is a latency feature with a hardware price; it belongs in the product discussion and in the cost table, separately.
If you read efficiency papers, the check is cheap and mechanical: find every place a sub-model is invoked, find whether its tokens and its time are in the same column as the main model's, and if they are not, ask what happens when you multiply the headline saving by the number of cards the setup needs. Here that one line turns 56% into 12% at best and −34% to −51% in the four settings where the compressor is the more expensive model.
Limits: no checkpoint has been released, so none of these numbers can be re-derived from weights — they are recomputed from the paper's own Tables 1–2 and its Python. The per-setting card layout is not fully stated, which is why the arithmetic above is given as a range and why the 12% figure is a best case rather than a measured one.