When Fancy Eviction Fails: Rethinking Cache Replacement for LLM Prefix Reuse
Researchers analyzed production traces to evaluate cache replacement algorithms for LLM prefix reuse, finding that sophisticated policies offer little benefit over standard LRU. The study attributes this to the regular pacing of active sessions, which makes recency a highly predictive metric for prefix reuse. The authors propose a new management strategy that retains recency as a foundation while adding specific mechanisms for demoting one-hit prefixes and handling compute-intensive misses.
- ▪The study evaluated 14 eviction algorithms using production traces from two companies under HBM-constrained and large memory-pool settings.
- ▪Sophisticated cache policies designed for traditional systems provide minimal improvement over LRU due to the structural nature of prefix reuse.
- ▪Prefix caching introduces challenges such as heavy-tailed session footprints and variable miss costs that increase with sequence length.
- ▪The authors introduce the compute-savings ratio and two offline oracles to quantify the effects of these caching challenges.
- ▪Effective cache management should combine recency-based retention with quick demotion for one-hit prefixes and compute-aware partial eviction.
Hacker News (AI / LLM) files mainly under ai. We currently carry 7,248 of its stories.
Story provenance
Source · retrieval · rights · ranking — open for full record
inspect →
Story provenance
Attribution is not the same as permission. This drawer separates discovery metadata, excerpts, WeSearch-generated summaries, reuse status, and whether the publisher receives the visit. Nothing here claims a legal grant the publisher has not made.
Record
| Original publisher | arXiv.org |
| Canonical URL | https://arxiv.org/abs/2609.28870 |
| Publication time | Thu, 01 Oct 2026 21:44:44 +0000 |
| Retrieval time | 2026-10-01T21:52:55.986Z |
| Last seen | 2026-10-01T21:52:55.986Z |
| Headline source | Publisher (no WeSearch rewrite) |
| Excerpt source | publisher body |
| Excerpt method | First ~120 words (~800 chars) of extracted publisher body, fair-use limited. |
| Summary | WeSearch · cerebras-chat (WeSearch summarizer) |
| Summary source text | contentText |
| Citation coverage | Summary is a WeSearch-generated derivative; primary citation is the original publisher URL. |
| Cluster | mD0bHDzNLVTh · 1 stories |
| Cluster logic | Grouped by semantic title/content similarity across sources within a rolling window. Same-publisher template collisions are excluded from coverage comparison. |
| Ranking reason | Story pages are not engagement-ranked. Hub feeds use recency, with optional source-diversified chronological ordering (cap consecutive stories per source). No personalized ranking. |
| Publisher visit | Yes — open original |
| Substitutes article? | No — link-out required for full text |
Rights status (four layers)
WeSearch handling by dimension
| Indexing | May the item be indexed (stored, ranked, made findable)? | Allowed |
| Snippet | May a short excerpt of the publisher's text be shown? | Allowed |
| AI summary | May WeSearch generate its own short summary of the article? | Limited |
| Retrieval / RAG | May the content be exposed for third-party retrieval-augmented generation? | Not asserted |
| Model training | May the content be used to train AI models? | Not asserted |
| Commercial reuse | May the content be reused commercially? | Not permitted |
Basis: Derived from the published RSS/Atom feed. Contact: [email protected]. Reviewed: 2026-07-24.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Distributed, Parallel, and Cluster Computing arXiv:2609.28870 (cs) [Submitted on 24 Sep 2026] Title:When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse Authors:Yiyu Liu, Minlan Yu, Juncheng Yang View a PDF of the paper titled When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse, by Yiyu Liu and 2 other authors View PDF HTML (experimental) Abstract:Long-running LLM applications repeatedly send growing context, making prefix caching critical for reducing prefill cost. Yet prefix-cache behavior under agentic workloads remains poorly understood. We study production traces from two companies and evaluate 14 eviction algorithms across HBM-constrained and large memory-pool settings.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv.org.
Discussion
0 commentsMore from arXiv.org
