Leto: Fast In-Place Recovery for LLM Training on Surviving Hardware

原文：Leto: Fast In-Place Recovery for LLM Training on Surviving Hardware

作者：Geon-Woo Kim, Joon Ha Kim, Daehyeok Kim

来源：arXiv cs.DC（分布式并行）

正文

Computer Science > Distributed, Parallel, and Cluster Computing

arXiv:2610.00687v1 (cs)

[Submitted on 30 Sep 2026]

Title:Leto: Fast In-Place Recovery for LLM Training on Surviving Hardware

Authors:Geon-Woo Kim, Joon Ha Kim, Daehyeok Kim

View a PDF of the paper titled Leto: Fast In-Place Recovery for LLM Training on Surviving Hardware, by Geon-Woo Kim and 2 other authors

View PDF

HTML (experimental)

Abstract:Hardware-operable failures (HOFs) interrupt large language model (LLM) training but permit recovery on the same hardware without reset, repair, or replacement. Existing recovery systems nevertheless reload checkpoints, recompute lost progress, and rebuild process state, idling GPUs that could otherwise continue training.

We present Leto, a fault-tolerant training system that leverages surviving hardware to enable efficient in-place recovery. Our key insight is that the state needed to resume training can be retained or prepared outside the active training process while remaining on the same hardware. Leto retains the working model state and the reusable process state, and preinitializes the remaining state in a shadow trainer. We devise two-tier erasure protection and chunk-level transactional updates to keep the retained model state recoverable and consistent, and reclaim the shadow state when active training needs its GPU memory. Evaluation on 6- and 72-GPU NVIDIA A100 clusters shows that Leto recovers 3.6–6.5$\times$ faster than the best-performing checkpointing baselines and improves productive training time by up to 13.7 percentage points. Large-scale simulation shows over 95% productive training time on a 131,072-GPU cluster.

Subjects:

Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG)

Cite as:

arXiv:2610.00687 [cs.DC]

(or

arXiv:2610.00687v1 [cs.DC] for this version)

https://doi.org/10.48550/arXiv.2610.00687

Focus to learn more

arXiv-issued DOI via DataCite

主题

分布式计算 · 机器学习

由「前沿雷达」于 2026-10-01 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。