ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference

原文：ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference

作者：Amir Ziashahabi, Hossein Entezari Zarch, Lei Gao, Murali Annavaram, Salman Avestimehr

来源：arXiv cs.DC（分布式并行）

正文

Computer Science > Machine Learning

arXiv:2609.17943v1 (cs)

[Submitted on 16 Sep 2026]

Title:ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference

Authors:Amir Ziashahabi, Hossein Entezari Zarch, Lei Gao, Murali Annavaram, Salman Avestimehr

View a PDF of the paper titled ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference, by Amir Ziashahabi and 4 other authors

View PDF

HTML (experimental)

Abstract:Long-context LLM inference is bottlenecked by attention, whose repeated KV-cache reads make decoding memory-bound. Self-speculative decoding alleviates this by drafting tokens with sparse attention and verifying them with full attention, but existing batched methods remain synchronized: all requests in a batch share a single draft-verify schedule, even though the optimal draft length varies widely across requests and changes dynamically within each request. We propose ASPIRE, a non-synchronized batched self-speculative decoding framework built on three components. First, a unified mixed forward allows drafting and verifying requests to coexist in the same batched forward pass, removing the need for global draft-verify phases. Second, a lightweight online speculation scheduler uses per-request acceptance-rate estimates and a batch-aware cost model to let each request independently choose when to verify. Third, an intra-draft refresh layer performs full attention at a single designated layer during drafting, updating the sparse context at every draft step to reduce staleness during drafting. Across three models and five reasoning and long-context benchmarks, ASPIRE achieves $1.70$-$4.58\times$ speedup in decoding throughput over autoregressive baselines and improves average speedup by approximately $27%$ over the strongest prior self-speculative baselines.

Comments:

Accepted to COLM 2026

Subjects:

Machine Learning (cs.LG); Computation and Language (cs.CL); Distributed, Parallel, and Cluster Computing (cs.DC)

Cite as:

arXiv:2609.17943 [cs.LG]

(or

arXiv:2609.17943v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2609.17943

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

机器学习 · 自然语言处理 · 分布式计算

由「前沿雷达」于 2026-09-16 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。