一句话：提出ETPR指标并构建自动稳定性治理框架，诊断和缓解大规模模型训练中的故障与慢速异常，提升集群训练生产力。
大规模基础模型训练常占用数千 GPU 数周至数月，但 ETTR 等指标只统计故障与重启损失，无法刻画“存活但进展缓慢”及隐性性能退化。论文提出可靠性导向的端到端生产力分解指标 ETPR，将损失归因于执行前损失、已分配未有效利用的计算时间和执行质量下降。作者在快手生产集群分析三个月内数千训练任务，发现 fail-stop、fail-hang 的诊断恢复及细微 fail-slow 异常是生产力下降主因。基于此，构建并部署自动稳定性治理框架：多阶段日志 fail-stop 诊断、时序栈不变性 fail-hang 定位、自适应 fail-slow/straggler 检测与分层恢复。实验表明其加快诊断、准确定位异常 rank、缓解静默减速并提升集群 ETPR。
Training large-scale foundation models often uses thousands of GPUs for weeks or months. In production clusters, training productivity is reduced not only by explicit failures but also by scheduling delays, repeated recoveries, alive-but-unproductive hangs, and performance degradation during seemingly normal execution. However, existing metrics such as Effective Training Time Ratio (ETTR) mainly account for time lost to failures and restarts, and do not directly capture jobs that remain alive but make slow progress. Prior diagnosis and mitigation efforts typically target specific failure modes, leading to case-by-case fixes and offering limited support for cluster-wide attribution of productivity loss or systematic reliability improvements. In this paper, we present the Effective Training Productivity Ratio (ETPR), a reliability-oriented decomposition of end-to-end training productivity to stabilize the large-scale machine learning (ML) systems. ETPR characterizes productivity loss by accounting for pre-execution loss, unproductive allocated compute time, and reduced execution quality. We conduct an empirical study of thousands of training jobs over three months on Kuaishou's production cluster. Our analysis shows that productivity is mainly degraded by diagnosis and recovery for fail-stop and fail-hang incidents, as well as subtle fail-slow anomalies that delay fault localization and worsen overall efficiency. Based on these findings, we build and deploy an automated stability governance framework that integrates: (i) a multi-stage log-based fail-stop diagnosis pipeline, (ii) temporal stack invariance analysis for fail-hang localization, (iii) an adaptive fail-slow detector with straggler detection, and (iv) hierarchical recovery strategies. Experiments show that this framework accelerates failure diagnosis, accurately localizes anomalous ranks, mitigates silent slowdowns, and improves cluster-wide ETPR in production.
@inproceedings{paperbot12413,
title = {Characterizing and Mitigating Productivity Loss in Large-Scale Model Training: An Empirical Study},
author = {Dinghao Xue and Yanxiang Chen and Junhong Liu and Yepeng Zhang and Guangsen Ni and Menghao Zhang and Tianyu Wo and Zheng Zheng and Chunming Hu and Jin Ouyang and Renyu Yang},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834462}
}