A 15-page paper shows long-context failure tracks answers that compete with the evidence, not length. Its sharpest result came from a generator that forgot to exclude the target key — and the corrected harness is in the repo, unrun.
Take Qwen2.5-1.5B at 32K tokens. Hold the position of the evidence fixed. Change only what surrounds it:
| What fills the context | Cost in nats | Accuracy |
|---|---|---|
| Neutral numeric text | 0.004 | 0.96–1.00 |
| The same text, masked from attention | 1.20 | drops sharply |
| Statements in the evidence template | 4.76 | 0.11–0.21 |
Thirty-two thousand tokens of filler cost nothing. Thirty-two thousand tokens of lookalikes cost most of the model's accuracy. Length never changed in the three rows — only content did.
The paper separates two kinds of lookalike. A statement can share the format of the evidence ("The secret code of the mountain is 9173") or it can share the key and hand it a rival value ("The secret code of the river is 5520"). Only the second one matters: explicit format-only distractors explain at most 0.3% of the variance in loss. Contexts with no rival lose 0.02–0.34 nats; contexts with at least one lose 1.38–3.24 nats, and each additional rival adds more — 0.03 → 3.45 nats from zero to six rivals at 2,048 tokens.
Inside the model, one number tracks it: the attention mass the final query token leaves on the evidence tokens. Across 126,473 trials, log(1/a*) explains 35–58% of the variance in loss; log length explains 2–8%. That gap survives from 8K to 32K tokens. This is the part of the paper that is genuinely new: the same trials measure behaviour, content and attention together, instead of inferring mechanism from accuracy curves.
Section 3 of the paper says, evenly: "Because keys are drawn uniformly, some competing statements share the queried key and assign it a different value." That is the whole mechanism of the headline result — and it happened by mistake. The repository README is blunter: in the distractor generator, keys were drawn from 51 nouns without excluding the target key, so roughly one filler sentence in 51 restated the queried key with a different value. "This was an accident."
The author put that in a warning block rather than a footnote, which is the right call and worth saying: the paper describes a deliberate contrast, and the artifact says the contrast was a collision.
Read carefully, the README also says what the paper cannot: format similarity without rivals was not tested in isolation. A corrected harness exists — it excludes the key and adds a controlled one-rival arm — but it has not been run, and no result from it appears anywhere in the repository. So the clean experiment that would separate "similar-looking text" from "contradicting text" is still open. The paper's own abstraction ("competition, not length") is supported for rivals; for mere similarity it is a prediction.
Every model here is 0.5B to 3B, and the task is a single synthetic key-value lookup. Nothing in the paper tests whether the same attention collapse explains failures on multi-document reasoning, code, or agent traces — the cases people actually mean when they say a model "forgot" something.
Attention and loss are read from the same forward pass, so the relationship is correlational; the README's own limits section says as much and notes no interventional test was run. Rival count also rises with length by construction, so length and conflict are not fully disentangled above 4K tokens: the "2K is enough to break it" reading only holds inside the controlled 512–2,048 grid.
And length is not harmless everywhere. The paper's replication shows masked filler costing 11–18 MMLU points at 30K tokens, and finds that whitespace filler is actively harmful — worse than masking on the retrieval task. The defensible claim is narrower than the headline: at fixed length, in retrieval, visible neutral text is nearly free, and rival answers are what break it.
Its methodological jab is the sharpest and the most contestable part: Du et al.'s visible control was whitespace, and whitespace is itself a harmful filler. That argument only lands if someone runs the visible-neutral control they did not run — which, again, is exactly the experiment sitting unrun in this repository.
If you are building retrieval, stop tuning chunk counts against token budgets and start measuring disagreement: how many passages in the context hand a different value to the same entity you are asking about. Deduplicate by entity, and treat "two documents disagree" as a first-class state to resolve before the model sees it, not as noise the model will average away.
If you are evaluating long context, report the rival count of your evaluation set. A benchmark whose contexts contain no conflicting values is measuring placement, not competition — and the difference is most of the failure.
Sources: arXiv:2611.04169 (v1, 3 Nov 2026) · code and README · 126K trial-level data. Numbers above are from the abstract, §4 and Table 2/3 as published; the "accident" paragraph quotes the repository README verbatim. No experiment was re-run for this note.