一句话：基于六个月生产故障实证研究，提出事件驱动的实时故障定位框架 Reflex，在高并发下实现 F1 0.88、诊断延迟 1.6 秒。
大规模 LLM 训练中基础设施故障频繁且代价高昂，而现有故障定位（定位故障节点）多为被动、依赖人工，严重制约训练吞吐。作者对生产环境中六个月收集的 2943 次 LLM 训练故障进行实证研究，得到三点关键认识：有效诊断需要融合全栈可见性与纵向性能剖析以建立时空上下文；诊断隐性故障需建模普遍存在的良性异常带来的复合影响；鲁棒检测需具备硬件感知能力以适应异构基础设施。据此提出 Reflex：通过纵向特征工程构建时空上下文以捕捉累积性退化；用可解释的事件归因解耦弱信号间依赖，从而识别隐性故障模式；并通过自适应噪声抑制与动态阈值保证异构环境下的鲁棒性。系统部署于 Platform-X，可精确高效地定位故障，在高并发下取得 0.88 的 F1 分数与 1.6 秒诊断延迟；作者还开源了匿名化的生产级告警轨迹数据集以推动社区研究。
Training Large Language Models (LLMs) at scale presents formidable reliability challenges, where frequent infrastructure failures incur prohibitive financial costs. Current fault localization (i.e., pinpointing faulty nodes) remains predominantly reactive and manual, severely constraining training throughput. We conduct a comprehensive empirical study of 2,943 LLM training failures collected over six months from a production environment. The analysis yields three critical insights: (1) effective diagnosis necessitates fusing full-stack visibility with longitudinal profiling to establish spatio-temporal context; (2) diagnosing implicit failures requires modeling the compound impact of pervasive benign anomalies; and (3) robust detection demands hardware-aware strategies adaptable to heterogeneous infrastructure. Guided by these insights, we propose Reflex, a real-time event-driven framework for automated fault localization. Reflex constructs a spatio-temporal context via longitudinal feature engineering to capture cumulative degradation. It employs interpretable event attribution to decode implicit failure patterns by disentangling dependencies among weak signals. Furthermore, the system ensures robustness across heterogeneous infrastructure through adaptive noise suppression and dynamic thresholding. Deployed in Platform-X, Reflex delivers precise and efficient fault localization, achieving an F1-score of 0.88 with a diagnosis latency of 1.6s under high concurrency. Finally, we open-source an anonymized dataset of production-grade alert traces to facilitate community research.
@inproceedings{DingZZWCYNSLLYL26,
title = {Reflex: Event-Driven Automated Fault Localization for Large-Scale LLM Training},
author = {Hua Ding and Yun Zhang and Bo Zhang and Wenxiao Wang and Libo Chen and Huan Yu and Zhe Nan and Zuquan Song and Weiqiang Lou and Gaohong Liu and Xi Yang and Yuhan Li and Qinlong Wang and Shuguang Wang and Wencong Xiao and Shenghong Li},
booktitle = {FSE},
year = {2026},
doi = {10.1145/3803437.3805245}
}