一句话：基于生产环境 LLM 训练失败的实证研究，提出日志分析框架 L4 自动诊断失败并定位故障节点。
LLM 训练需大量算力与时间，失败不可避免且代价高昂。本文首次对生产平台 Platform-X 在 2023 年 5 月至 2024 年 4 月间的 428 次 LLM 训练失败报告进行实证研究，发现硬件与用户故障是主要根因，现有诊断高度依赖训练日志，但现有日志诊断方法难以处理 LLM 训练日志。作者识别出 LLM 训练日志的跨作业、空间与时间三种独特模式，提出 L4 框架，自动从海量日志中提取指示失败的信息（日志事件、节点、阶段与迭代），减少人工并加速恢复。在真实数据集上，L4 在识别失败指示日志与定位故障节点上优于现有方法，并已在 Platform-X 部署，证明其有效性与效率。
As Large Language Models (LLMs) show their capabilities across various applications, training customized LLMs has become essential for modern enterprises. However, due to the complexity of LLM training, which requires massive computational resources and extensive training time, failures are inevitable during the training process. These failures result in considerable waste of resource and time, highlighting the critical need for effective and efficient failure diagnosis to reduce the cost of LLM training. In this paper, we present the first empirical study on the failure reports of 428 LLM training failures in our production Platform-X between May 2023 and April 2024. Our study reveals that hardware and user faults are the predominant root causes, and current diagnosis processes rely heavily on training logs. Unfortunately, existing log-based diagnostic methods fall short in handling LLM training logs. Considering the unique features of LLM training, we identify three distinct patterns of LLM training logs: cross-job, spatial, and temporal patterns. We then introduce our Log-based Large-scale LLM training failure diagnosis framework, L4, which can automatically extract failure-indicating information (i.e., log events, nodes, stages, and iterations) from extensive training logs, thereby reducing manual effort and facilitating failure recovery. Experimental results on real-world datasets show that L4 outperforms existing approaches in identifying failure-indicating logs and localizing faulty nodes. Furthermore, L4 has been applied in Platform-X and demonstrated its effectiveness in enabling accurate and efficient failure diagnosis.
@inproceedings{abs-2503-20263,
title = {L4: Diagnosing Large-scale LLM Training Failures via Automated Log Analysis},
author = {Zhihan Jiang and Junjie Huang and Guangba Yu and Zhuangbin Chen and Yichen Li and Renyi Zhong and Cong Feng and Yongqiang Yang and Zengyin Yang and Michael Lyu},
booktitle = {FSE},
year = {2025},
doi = {10.1145/3696630.3728531}
}