Characterizing High Bandwidth Flash for LLM Serving

原文：Characterizing High Bandwidth Flash for LLM Serving

作者：Zack Yu, Chloe Wong, Coleman Hooper, Minjae Lee, Wonjun Kang, Youngjin Cho, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami

来源：arXiv cs.DC（分布式并行）

正文

Computer Science > Machine Learning

arXiv:2609.39131v1 (cs)

[Submitted on 30 Sep 2026]

Title:Characterizing High Bandwidth Flash for LLM Serving

Authors:Zack Yu, Chloe Wong, Coleman Hooper, Minjae Lee, Wonjun Kang, Youngjin Cho, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami

View a PDF of the paper titled Characterizing High Bandwidth Flash for LLM Serving, by Zack Yu and 9 other authors

View PDF

HTML (experimental)

Abstract:Large language model (LLM) serving requires substantial memory to store model weights and KV caches. As models grow larger and contexts become longer, memory capacity and bandwidth increasingly become bottlenecks for serving performance. Agentic workloads compound this pressure through repeated interactions over growing contexts, making it increasingly important to retain KV state for reuse. High-bandwidth flash (HBF) offers a way to expand accelerator memory capacity for large language model (LLM) serving, but its access costs and limited write endurance complicate its use. We evaluate HBF for high-throughput agentic serving across system design and scheduling choices to understand when additional capacity improves serving performance and energy efficiency. We introduce an HBM-HBF-host hierarchical storage system and buffered cache-aware scheduling, and use trace-driven simulations to analyze their effects on performance, energy consumption, and HBF write lifetime. Across the evaluated workloads, the fastest HBF-augmented systems reduce completion time by 36.1-87.0% relative to HBM-only systems. Modeled energy savings reach 55.8%, although HBF increases energy consumption on some light workloads. Buffered cache-aware scheduling extends estimated HBF write lifetime from 4.77 to 14.82 years in the evaluated configuration. These results demonstrate the importance of coordinating data placement and scheduling to improve serving efficiency while sustaining a practical HBF write lifetime.

Subjects:

Machine Learning (cs.LG); Hardware Architecture (cs.AR); Distributed, Parallel, and Cluster Computing (cs.DC)

Cite as:

arXiv:2609.39131 [cs.LG]

(or

arXiv:2609.39131v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2609.39131

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

机器学习 · cs.AR · 分布式计算

由「前沿雷达」于 2026-09-30 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。