Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs

原文：Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs

作者：Zhengxiang Huang, Shengheng Chen, Chaoyue Niu, Yujie Sun, Zhaode Wang, Zeyu Zhao, Chengfei Lv, Fan Wu, Guihai Chen

来源：arXiv cs.DB（数据库）

正文

Computer Science > Operating Systems

arXiv:2609.34727v1 (cs)

[Submitted on 28 Sep 2026]

Title:Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs

Authors:Zhengxiang Huang, Shengheng Chen, Chaoyue Niu, Yujie Sun, Zhaode Wang, Zeyu Zhao, Chengfei Lv, Fan Wu, Guihai Chen

View a PDF of the paper titled Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs, by Zhengxiang Huang and 8 other authors

View PDF

HTML (experimental)

Abstract:On-device large language model (LLM) serving is a cornerstone of local-first personal intelligence, offering users data sovereignty, strong privacy guarantees, and freedom from cloud API latency and cost. Although KV caching is widely used to reduce latency in long-context inference, existing designs were primarily optimized for cloud GPUs with dynamic execution environments and abundant memory bandwidth. These architectural assumptions do not hold on mobile NPUs, where computation graphs must be statically compiled and both memory capacity and I/O bandwidth are severely constrained. In this work, we present a compute-storage co-design for mobile-centric prefix and non-prefix KV reuse. We first propose an intra-graph mechanism that maps selective KV recomputation onto static NPU graphs, reconciling algorithmic dynamicity with NPU staticity. We further develop an inter-graph scheduler to optimize chunk merging and minimize padding with dynamic programming. To address mobile bandwidth limitations, we introduce a hierarchical KV manager featuring a tree-hash-semantic hybrid structure, along with cost-aware prefetching and eviction policies. We also build a two-dimensional pipeline that overlaps KV loading, rerotation, and storage with NPU execution, hiding data-movement latency. Experiments across representative on-device workloads and LLMs show that our design reduces time-to-first-token (TTFT) by $40-60%$ compared with no reuse and prefix-only caching.

Comments:

Accepted at EuroSys 2027 (Spring Cycle)

Subjects:

Operating Systems (cs.OS); Artificial Intelligence (cs.AI); Databases (cs.DB)

Cite as:

arXiv:2609.34727 [cs.OS]

(or

arXiv:2609.34727v1 [cs.OS] for this version)

https://doi.org/10.48550/arXiv.2609.34727

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

cs.OS · 人工智能 · 数据库

由「前沿雷达」于 2026-09-28 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。