BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models

原文：BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models

作者：Ning Zhang, Shihao Wang, Jinrui Zhang, Chaodong Xiao, Lei Zhang

来源：arXiv cs.AI（人工智能）

正文

Computer Science > Computation and Language

arXiv:2610.28500v1 (cs)

[Submitted on 20 Oct 2026]

Title:BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models

Authors:Ning Zhang, Shihao Wang, Jinrui Zhang, Chaodong Xiao, Lei Zhang

View a PDF of the paper titled BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models, by Ning Zhang and 4 other authors

View PDF

HTML (experimental)

Abstract:For efficient deployment of large language models (LLMs), methods of ultra-low-bit backbones reduce weight storage, memory traffic, and linear-layer execution cost, while multi-token prediction (MTP) methods reduce sequential decoding steps by predicting multiple future tokens in parallel. Although complementary, the direct combination of these methods cannot accelerate decoding at comparable quality: masked drafting lacks intermediate token input supplied in next-token training, while wider computation and causal key-value (KV) updates can offset the savings from fewer steps. To address these challenges, we propose BitMTP, an algorithm-system co-design framework for efficient MTP on ultra-low-bit LLM backbones. A joint next-token prediction and MTP objective equips a single model with both autoregressive (AR) generation and block-causal drafting. Confidence-only decoding, speculative-only decoding, and hybrid decoding provide different quality-throughput operating points without using separate draft and verifier models. Accepted-prefix alignment keeps the committed prefix, materialized causal KV boundary, and CUDA Graph replay state consistent for continued generation. BitMTP uses this contract to select width-specialized low-bit execution and reuse captured graphs over shared KV storage. On NVIDIA H20, BitMTP with hybrid decoding achieves five-task mean speedups of 2.2x on BitNet b1.58-2B and 2.1-2.4x on 1.7B-4B Bonsai models at comparable task quality. On RTX 3090 and Jetson Orin, it further achieves up to 2.0x in five-task mean throughput and 3.3x on MBPP+, respectively. These results demonstrate that BitMTP accelerates ultra-low-bit LLM generation across backbones, model sizes, and GPU types. The source code is released at this https URL

Comments:

Accepted at NeurIPS 2026. 16 pages, 3 figures. Code: this https URL

Subjects:

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG)

Cite as:

arXiv:2610.28500 [cs.CL]

(or

arXiv:2610.28500v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2610.28500

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

自然语言处理 · 人工智能 · 分布式计算 · 机器学习

由「前沿雷达」于 2026-10-21 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。