一句话：提出 KAIOPS，一个用于大规模 AI 训练集群的端到端多模态 AIOps 平台，实现异常检测、根因分析与修复方案生成。
大规模 AI 训练平台中异常频发，传统异常检测难以应对复杂时序特征与极端类别不平衡，且缺少自动化根因分析和后续修复。KAIOPS 是快手提出的端到端多模态 AIOps 平台，采用时序上下文编码机制捕获故障演化中的长期趋势和关键时序上下文，并设计动态类别加权损失提升检测性能；进一步结合知识图谱与 LLM 实现自动根因分析和可执行解决方案生成。基于快手生产级训练集群数据的实验表明方法性能优越，平台已在含超过 10,000 块 GPU 的测试与生产环境中部署，加速了工业级模型训练与服务的可靠性保障。
The resilience of large-scale AI training platforms are fundamental to enabling contemporary AI innovation and business development. However, with the rapid increase in the scale and complexity of AI model training tasks, anomalies become the norm rather than the exception at scale. Failing to handle them properly may lead to enormous resource waste and prolonged development cycles. Traditional anomaly detection methods struggle to tackle the complex temporal characteristics and extreme class imbalance inherently manifesting in training tasks, and fall short in automated solution to root cause analysis and the follow-up remediation. This paper proposes KAIOPS, an end-to-end automated platform solution for handling anomalies and engineering experience of daily operational maintenance for large-scale AI training clusters at Kuaishou. KAIOPS employs a Temporal Context Encoding mechanism to precisely capture and encode long-term trends and critical temporal context information within fault evolution. The detection model elaborates a dynamic class-weighted loss function for enhancing the detection performance. To deliver a complete end-to-end intelligent processing pipeline, KAIOPS further leverages knowledge graph and LLMs for automated root cause analysis and actionable solution generation. Extensive experiments, on the basis of data collected from Kuaishou’s production-grade training clusters, show the superior performance of our proposed approach. KAIOPS has been deployed in Kuaishou, in both testbed and production grade environments, consisting of with over 10,000 GPUs, and accelerate the reliability assurance for industry-scale model training and serving.
DOI 原文 ·
@inproceedings{paperbot9204,
title = {KAIOPS: A Platform Solution of End-to-End Multi-Modal AIOps for AI Training at Scale},
author = {Zeying Wang and Junhong Liu and Penghao Zhang and Xiaoyang Sun and Xu Wang and Tianyu Wo and Chunming Hu and Chengru Song and Jin Ouyang and Renyu Yang},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00263}
}