LayerCheck: Adaptive Layer-wise Checkpointing for Large Language Model Post-training

原文：LayerCheck: Adaptive Layer-wise Checkpointing for Large Language Model Post-training

作者：Minqiu Sun, Xin Huang, Luanzheng Guo, Nathan R. Tallent, Kento Sato, Dong Dai

来源：arXiv cs.DC（分布式并行）

正文

Computer Science > Distributed, Parallel, and Cluster Computing

arXiv:2609.27193v1 (cs)

[Submitted on 23 Sep 2026]

Title:LayerCheck: Adaptive Layer-wise Checkpointing for Large Language Model Post-training

Authors:Minqiu Sun, Xin Huang, Luanzheng Guo, Nathan R. Tallent, Kento Sato, Dong Dai

View a PDF of the paper titled LayerCheck: Adaptive Layer-wise Checkpointing for Large Language Model Post-training, by Minqiu Sun and 5 other authors

View PDF

HTML (experimental)

Abstract:With the rising computational and monetary costs of training large language models (LLMs), checkpointing—periodically storing model states for recovery—becomes essential for fault tolerance. Conventional checkpointing entails a severe trade-off between checkpoint frequency (I/O overhead) and computational recovery (recovery time). State-of-the-art approaches mitigate this cost through pipelining checkpoint I/Os, differential checkpointing, or in-memory persistence, yet none leverage the distinct characteristics of LLM training dynamics, where model weight updates are non-uniformly distributed across transformer layers. This observation implies that saving all weights each time might not be efficient. Inspired by this observation, we present LayerCheck, a layer-wise adaptive checkpointing framework that selectively persists layers whose updates exceed a threshold. This design avoids periodic I/O bursts by distributing layer-wise checkpoint writes over time, resulting in smoother and more balanced I/O profiles. Upon recovery, LayerCheck reconstructs a mixed-timestamp composite model state by aggregating the most recently persisted versions of each layer together with their matching optimizer states. Under a bounded per-layer staleness guard, this introduces a controlled perturbation: under standard Adam assumptions it adds a bounded staleness term, and empirically the post-restart loss deviates from the failure-free trajectory by at most 0.54%. Empirical results on multiple open-source LLMs with different datasets further demonstrate that recovered models preserve the original convergence behavior and accuracy while substantially reducing checkpoint overheads. Specifically, LayerCheck achieves up to 22.6x reduction in total checkpoint size and 1.31x reduction in end-to-end training time compared to state-of-the-art systems, significantly lowering the cost of checkpointing.

Comments:

Accepted to the 2026 IEEE International Conference on Cluster Computing (CLUSTER 2026). 19 pages, 8 figures, 3 tables

Subjects:

Distributed, Parallel, and Cluster Computing (cs.DC)

Cite as:

arXiv:2609.27193 [cs.DC]

(or

arXiv:2609.27193v1 [cs.DC] for this version)

https://doi.org/10.48550/arXiv.2609.27193

Focus to learn more

arXiv-issued DOI via DataCite

主题

分布式计算

由「前沿雷达」于 2026-09-23 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。