DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized Batching

原文：DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized Batching

作者：Jingpo Xu, Paul Joe Maliakel, Ivona Brandic, Shashikant Ilager

来源：arXiv cs.DC（分布式并行）

正文

Computer Science > Distributed, Parallel, and Cluster Computing

arXiv:2610.08268v1 (cs)

[Submitted on 6 Oct 2026]

Title:DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized Batching

Authors:Jingpo Xu, Paul Joe Maliakel, Ivona Brandic, Shashikant Ilager

View a PDF of the paper titled DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized Batching, by Jingpo Xu and 3 other authors

View PDF

HTML (experimental)

Abstract:Pervasive intelligent applications are increasingly deployed on mobile and Internet of Things (IoT) edge devices. Consequently, Large Language Models (LLMs) are increasingly used to support these applications. Yet, due to their high resource demands, LLMs are mostly deployed in the cloud. Layer-wise edge-cloud inference lets resource-constrained edge devices contribute computation to LLMs they cannot host in full. However, heterogeneous split points introduce two coupled inefficiencies. First, edge execution and communication create idle gaps between cloud invocations. Second, requests arriving at different model depths cannot be conventionally batched. We present DySCo, a collaborative runtime that keeps KV caches local and introduces dyForward, a model-aware layer-range executor that runs configurable contiguous layer ranges from resident model shards without reloading weights. For multi-edge serving settings, we introduce depth-synchronized batching (DSB), which advances heterogeneous requests to the deepest cut and batches their common suffix. Experiments across heterogeneous devices, two model families, and local and wide-area links show that idle gaps increase the latency of subsequent GPU forward calls even when waiting time is excluded, adding up to 25 ms of additional cloud-side suffix latency per decoding step in our measurements. At an average concurrency of eight, DSB improves throughput by 275% over FIFO, 48% over exact-match batching, and 79% over round-robin interleaving while reducing mean per-session latency. Together, these results show that requests with different edge-cloud splits can reuse resident cloud weights and share batched suffix computation. The artifact repository for this work is publicly available at: this https URL

Comments:

article under submission

Subjects:

Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Systems and Control (eess.SY)

Cite as:

arXiv:2610.08268 [cs.DC]

(or

arXiv:2610.08268v1 [cs.DC] for this version)

https://doi.org/10.48550/arXiv.2610.08268

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

分布式计算 · 人工智能 · 机器学习 · eess.SY

由「前沿雷达」于 2026-10-06 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。