一句话：提出 Pulse，通过网卡微秒级 RDMA 流量测量实现非侵入式细粒度 LLM 训练监控与通信算子异常定位。
LLM 训练持续时间长、规模大，异常会因同步特性产生级联效应，诊断困难；现有方法依赖代码插桩收集通信算子信息，粒度粗且需修改训练代码或通信库。Pulse 在网卡上执行微秒级 RDMA 流量测量，并将流级测量转换为通信算子级测量，从而实现细粒度、非侵入且易部署的监控。在 64 卡 H200 测试平台上，12 个常见故障场景中 Pulse 在 10 个中实现机器级定位，现有方法仅成功 4 个且误诊 2 个；同时达到超过 90% 精度、100% 召回，每网卡支持最多 2000 个并发 RDMA 流测量，训练性能开销可忽略。
Large language model (LLM) training is prone to anomalies due to its long duration and large scale, which can lead to significant performance degradation or even training crashes. Due to the synchronization nature of LLM training, anomalies exhibit the cascading effect, making their diagnosis challenging. Existing approaches rely on collecting communication operator information via code instrumentation, which yields only coarse-grained monitoring data and requires modifications to training code or communication libraries. We propose Pulse, a fine-grained, non-intrusive, and easy-to-deploy monitoring system. Our key idea is to enable fine-grained monitoring via traffic measurement. Pulse conducts microsecond-level RDMA traffic measurement on NICs, and transforms flow-level measurements into communication operator measurements, thereby enabling fine-grained and non-intrusive monitoring. We deploy Pulse on a testbed with 64 H200 GPUs and evaluate its anomaly localization capability under common failure scenarios. Pulse achieves machine-level localization in 10 out of 12 scenarios, while existing methods succeed in only 4 and even misdiagnose 2 of the remaining scenarios. Additionally, Pulse achieves over 90% precision and 100% recall, supports up to 2000 concurrent RDMA flow measurements per NIC, and imposes negligible overhead on training performance, making it a practical solution for real-world LLM training environments.
@inproceedings{XiaoZ00DH0C026,
title = {Fine-grained and Non-intrusive LLM Training Monitoring via Microsecond-level Traffic Measurement},
author = {Yibo Xiao and Hao Zheng and Haifeng Sun and Qingkai Meng and Jiong Duan and Xiaohe Hu and Rong Gu and Guihai Chen and Chen Tian},
booktitle = {ASPLOS},
year = {2026},
doi = {10.1145/3779212.3790163}
}