The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache. Existing eviction or compression methods reduce cost but rely on heuristics, such as recency or past attention scores, which serve only as indirect proxies for a token’s future utility and introduce computational overhead. We reframe KV cache eviction as a reinforcement learning (RL) problem: learning to rank tokens by their predicted usefulness for future decoding. To this end, we introduce KV Policy (KVP), a framework of… Read More
Learning to Evict from Key-Value Cache
The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache. Existing eviction or compression methods reduce cost but rely on heuristics, such as recency or past attention scores, which serve only as indirect proxies for a token’s future utility and introduce computational overhead. We reframe KV cache eviction as a reinforcement learning (RL) problem: learning to rank tokens by their predicted usefulness for future decoding. To this end, we introduce KV Policy (KVP), a framework of…
Related Posts
Speech Foundation Models Generalize to Time Series Tasks from Wearable Sensor Data
This paper was accepted at the Learning from Time Series for Health workshop at NeurIPS…
How generative AI can help scientists synthesize complex materials
MIT researchers’ DiffSyn model offers recipes for synthesizing new materials, enabling faster experimentation and a…
MCPs for Developers Who Think They Don’t Need MCPs
The following article originally appeared on Block’s blog and is being republished here with the author’s permission.…