CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning

原文：CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning

作者：Yafei Zhang, Songshuo Lu, Sicong Liao, Zhi Chen, Yaohua Tang

来源：arXiv cs.AI（人工智能）

正文

Computer Science > Machine Learning

arXiv:2610.02039v1 (cs)

[Submitted on 1 Oct 2026]

Title:CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning

Authors:Yafei Zhang, Songshuo Lu, Sicong Liao, Zhi Chen, Yaohua Tang

View a PDF of the paper titled CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning, by Yafei Zhang and 4 other authors

View PDF

HTML (experimental)

Abstract:Recent years have witnessed the rapid adoption of reinforcement learning (RL) in large language model (LLM) post-training, with substantial gains in mathematical reasoning and code generation. In practical systems, however, policy updates and differences between rollout and training engines can make sampled responses off-policy. Sequence-level masking addresses this mismatch by deciding whether an entire response should contribute to optimization. A common masking rule uses the length-normalized geometric mean of sampled token probability ratios. Its signed log-ratios can cancel across positions, concealing substantial bidirectional policy drift. We propose \emph{Cancellation-Aware Response Masking} (CARM), a sequence-level mask that takes the absolute value of each token log-ratio before averaging, preventing opposing probability changes from canceling. We prove that accepted responses satisfy a joint bound on the fraction of sampled-token ratios outside a prescribed band and their mean log-distance beyond its boundaries. Experiments on mathematical reasoning and code generation show that CARM improves mean@16 averaged over AIME 2024/2025/2026 and BeyondAIME by up to $3.13$ percentage points over geometric-mean masking, and increases average pass@1 across four code benchmarks by $2.88$ points over the strongest evaluated baseline. These findings support CARM as a theoretically grounded and effective method for response-level off-policy control in LLM reinforcement learning.

Comments:

28 pages, 11 figures, 5 tables

Subjects:

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)

Cite as:

arXiv:2610.02039 [cs.LG]

(or

arXiv:2610.02039v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.02039

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

机器学习 · 人工智能 · 自然语言处理

由「前沿雷达」于 2026-10-02 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。