Cache hit rates of Inference are more meaningful than the headline costs
The article discusses the significance of cache hit rates in the context of long multi-turn conversations with language models. It highlights that agents push the entire conversation history into context, making cache efficiency crucial for cost management. An analysis of over 60 providers reveals that cache hit rates are often overlooked yet play a vital role in determining overall processing costs.
- ▪Agents in multi-turn conversations are extremely read heavy due to the full conversation history being pushed into context every turn.
- ▪The analysis utilized data from over 60 providers and 398 data points sourced from openrouter.ai model pages.
- ▪Cache hit rates significantly impact the costs associated with processing input tokens in long conversations.
Opening excerpt (first ~120 words) tap to expand
.pct-high { color: #10b981; font-weight: 700; } .pct-mid { color: #f59e0b; font-weight: 700; } .pct-low { color: #ef4444; font-weight: 700; } .callout { @apply p-6 rounded-2xl border mb-8; } Tl;Dr: Agents push the full conversation history into context every turn; hence, over a large number of turns, they are extremely read heavy, which in turn is why cache hit rates are an important factor. This post is an analysis of 60+ providers and their cache hit rates using 398 data points. All data sourced from openrouter.ai model pages. Agentic workflows are different from most human-LLM conversations in one key characteristic: the number of turns on average are far higher. Context processing over multi-turn conversation grows quadratically.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Dirac.
Discussion
0 commentsMore from Dirac
