SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving

原文：SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving

作者：Chuan Liu, Shuoming Zhang, Zhicheng Li, Qianqi Sun, Ruiyuan Xu, Qiuchu Yu, Xiyu Shi, Huimin Cui, Jiacheng Zhao

来源：arXiv cs.DC（分布式并行）

正文

Computer Science > Distributed, Parallel, and Cluster Computing

arXiv:2609.37626v1 (cs)

[Submitted on 29 Sep 2026]

Title:SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving

Authors:Chuan Liu, Shuoming Zhang, Zhicheng Li, Qianqi Sun, Ruiyuan Xu, Qiuchu Yu, Xiyu Shi, Huimin Cui, Jiacheng Zhao

View a PDF of the paper titled SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving, by Chuan Liu and 8 other authors

View PDF

HTML (experimental)

Abstract:No single way of parallelizing attention serves large language models well under all loads. Low concurrency favors tensor parallelism, many independent requests favor data-parallel attention, and long prompts favor context parallelism. Reasoning, agentic, and RL-rollout workloads make a fixed choice untenable: a batch that begins as many short requests ends as a few very long ones, so the best layout changes while the same requests run. Serving engines nevertheless fix one layout at launch, because changing it has meant draining requests and restarting workers. We present SPLASH, a serving system that switches the parallel layout of attention while requests are running. It builds on one observation: modern attention, with few or no KV heads, decouples where a request’s KV cache lives from how attention weights are sharded. This has two consequences. First, layouts differ only in who owns the weights and the cache, and most of that state already sits where the next layout needs it; SPLASH reuses it, moves the rest in the background of ongoing inference, and hands off at a batch boundary, making a switch nearly free: its median overhead is under 0.51% of the step it runs in. Second, the decoupling exposes a layout that existing engines lack: Decoupled Ownership Parallelism (DOP) shards attention weights as tensor parallelism does while keeping each request’s cache on a single owner as data-parallel attention does. DOP replicates neither, offers 27-60% more KV capacity than data-parallel attention, and gives the scheduler a choice when KV memory limits admission. A transition-aware scheduler follows the best of the four layouts as load changes. On B200 GPUs serving GLM-5.3, SPLASH improves end-to-end serving throughput by 1.3-1.73x over fixed-layout deployments, and the same layout regimes appear with DeepSeek-V3.2 on H200 and GLM-5.3-Flash on DCU.

Comments:

28 pages, 11 figures, 8 tables. Code: this https URL

Subjects:

Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI)

Cite as:

arXiv:2609.37626 [cs.DC]

(or

arXiv:2609.37626v1 [cs.DC] for this version)

https://doi.org/10.48550/arXiv.2609.37626

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

分布式计算 · 人工智能

由「前沿雷达」于 2026-09-29 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。