TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash

原文：TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash

作者：Jay H. Park, Hyungjun Kim, Dong Kim

来源：arXiv cs.DC（分布式并行）

正文

Computer Science > Distributed, Parallel, and Cluster Computing

arXiv:2609.35065v1 (cs)

[Submitted on 28 Sep 2026]

Title:TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash

Authors:Jay H. Park, Hyungjun Kim, Dong Kim

View a PDF of the paper titled TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash, by Jay H. Park and 2 other authors

View PDF

HTML (experimental)

Abstract:Reusable prefix key-value (KV) caches can outgrow GPU memory in large language model (LLM) serving. A memory-semantic flash hierarchy offers SSD-backed capacity with a limited fast tier, but a logical KV hit is not necessarily ready for GPU retrieval. Demand staging exposes SSD latency, whereas immediate staging can reserve fast-tier capacity long before retrieval begins. We present TempoKV, a timing-aware resource-commitment layer that separates early knowledge of reuse from the acquisition of staging resources. It records reusable-KV hits as metadata-only claims and requests commitment when the runtime-estimated time until retrieval falls to the storage-estimated time needed to make KV resident and protected against eviction. These estimates adapt to runtime progress and staging state, while commitment remains subject to available protected capacity. We implement TempoKV in vLLM and LMCache on an SSD-backed CXL memory device without changing request scheduling. Across two models and three prefix cache ratios, TempoKV reduces protected fast-tier byte-time per request by 63-91% versus immediate staging while retaining much of the serving benefit of advance staging. In a fast-tier capacity sweep, output throughput and p95 time to first token (TTFT) remain nearly unchanged as capacity decreases from 100 to 25 GiB. Compared with unmodified LMCache’s Device-DAX L1 configuration, TempoKV reduces p95 TTFT by up to 48.0% and increases output throughput by up to 27.8%.

Subjects:

Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

Cite as:

arXiv:2609.35065 [cs.DC]

(or

arXiv:2609.35065v1 [cs.DC] for this version)

https://doi.org/10.48550/arXiv.2609.35065

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

分布式计算 · 人工智能 · 机器学习

由「前沿雷达」于 2026-09-28 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。