作者：Tianrui Liu, Xiaohai Hu
来源：arXiv cs.DC（分布式并行）
正文
Computer Science > Performance
arXiv:2609.20874v1 (cs)
[Submitted on 16 Sep 2026]
Title:Decomposing Predictive Kubernetes Autoscaling for Large Language Model Serving Under Long Startup Delays
Authors:Tianrui Liu, Xiaohai Hu
View a PDF of the paper titled Decomposing Predictive Kubernetes Autoscaling for Large Language Model Serving Under Long Startup Delays, by Tianrui Liu and 1 other authors
View PDF
HTML (experimental)
Abstract:Large language model (LLM) inference deployed on Kubernetes faces an autoscaling challenge that conventional web services do not: new serving replicas take two to ten minutes to start because multi-gigabyte model weights must be loaded, which makes purely reactive scaling structurally late. We ask a sharp question: among the components of a predictive autoscaler, which ones actually matter under such long actuation delays? We answer it by decomposing predictive autoscaling into four factors—token-aware demand tracking, startup-delay lookahead, a bounded uncertainty margin, and plant-state observation—and measuring each factor’s contribution in isolation on production-derived heavy-tailed workloads generated by ServeGen. Our main finding is that a simple exponentially weighted moving average (EWMA) predictor with delay-aware lookahead and an upper confidence bound (UCB) margin captures most of the benefit, reducing time-to-first-token (TTFT) service-level-objective (SLO) violations from 53% (reactive, queries-per-second based) to 0.5% across five random seeds; lookahead alone is the single largest factor, a 14$\times$ reduction. Kalman filter variants do not consistently improve the cost–SLO tradeoff. Controlled experiments isolate the reason token granularity is necessary: context length, through key–value cache pressure, degrades TTFT far more than request rate at matched throughput. Finally, a validation on a real Kubernetes cluster (Qwen2.5-7B, A100, vLLM) confirms the central mechanism: a delay-aware lookahead controller cuts TTFT violations from 63.5% to 3.7% relative to reactive KEDA scaling. We distinguish throughout which findings are specific to Kubernetes actuation and which are general to LLM serving.
Comments:
accepted by CloudCom 2026
Subjects:
Performance (cs.PF); Distributed, Parallel, and Cluster Computing (cs.DC)
Cite as:
arXiv:2609.20874 [cs.PF]
(or
arXiv:2609.20874v1 [cs.PF] for this version)
https://doi.org/10.48550/arXiv.2609.20874
Focus to learn more
arXiv-issued DOI via DataCite
主题
cs.PF · 分布式计算
由「前沿雷达」于 2026-09-16 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。
评论
加载中…