$S^3$: Spectral Null-Space Swap Makes Reasoning Models Efficient

原文：$S^3$: Spectral Null-Space Swap Makes Reasoning Models Efficient

作者：Hongbo Ma, Sansheng Cao, Jiajun Fan, Bangji Yang, Ge Liu

来源：arXiv cs.AI（人工智能）

正文

Computer Science > Machine Learning

arXiv:2609.37976v1 (cs)

[Submitted on 29 Sep 2026]

Title:$S^3$: Spectral Null-Space Swap Makes Reasoning Models Efficient

Authors:Hongbo Ma, Sansheng Cao, Jiajun Fan, Bangji Yang, Ge Liu

View a PDF of the paper titled $S^3$: Spectral Null-Space Swap Makes Reasoning Models Efficient, by Hongbo Ma and 4 other authors

View PDF

HTML (experimental)

Abstract:LLMs trained with Chain-of-thought excel in reasoning capability, but often come with excessive token cost. We find that the core of reasoning capacity lies in the Thinking model’s weight component within the null space of a projection defined by the corresponding Non-thinking model’s dominant singular directions, and removing the subspace component can largely improve reasoning efficiency without hurting the accuracy gained during thinking-mode post-training. Unlike existing efforts that mostly operate within the dominant subspace, we are the first to unveil the critical role of the null space and harness it for model optimization. Motivated by this finding, we propose Spectral Null-Space Swap ($S^3$), a training-free composition of paired Non-thinking and Thinking checkpoints. Our method keeps the Non-thinking model inside its own dominant subspace and takes the Thinking checkpoint outside it, improving reasoning efficiency while maintaining accuracy. We extensively evaluate $S^3$ on 2B-30B dense and mixture-of-experts (MoE) architectures spanning 28 evaluation environments across mathematical, multimodal, and audio reasoning domains. $S^3$ establishes new empirical Pareto Frontiers among training-free model composition strategies: across all settings, it reduces inference token overhead by an average of 27.4% compared to full Thinking models while simultaneously improving overall task accuracy by 1.0 percentage point (e.g., yielding +8.3% accuracy on HMMT25 alongside a 33.0% token speedup). We further use attention entropy for explanation and find that the retained component produces more concentrated attention, and we use a simplified analytical model about optimization to demonstrate why null-space can effectively reduce attention entropy, thereby improving the efficiency of reasoning.

Comments:

44 pages, 9 figures, 29 tables

Subjects:

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)

Cite as:

arXiv:2609.37976 [cs.LG]

(or

arXiv:2609.37976v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2609.37976

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

机器学习 · 人工智能 · 自然语言处理 · 计算机视觉

由「前沿雷达」于 2026-09-30 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。