一句话:提出基于图的在线系统可用性事件表示、GNN 检测与 PageRank 根因定位方法,并在微信系统验证部署。
大规模在线系统中,可用性问题常被海量告警淹没,且会沿服务依赖产生级联效应,人工难以识别真正 incident 并定位根因。本文提出自动分析可用性问题级联效应的方法:从问题症状与受影响服务属性中抽取基于图的 issue 表示;基于该表示训练图神经网络进行 incident 检测;对检测出的事件,设计带有灵活转移矩阵的 PageRank 算法定位根因。作者使用来自微信在线服务系统的真实数据评估,结果证实方法有效,并且该方法已在公司部署,缓解了运维人员面对大量问题和相关告警信号时的负担。
With the ever increasing scale and complexity of online systems, incidents are gradually becoming commonplace. Without appropriate handling, they can seriously harm the system availability. However, in large-scale online systems, these incidents are usually drowning in a slew of issues (i.e., something abnormal, while not necessarily an incident), rendering them difficult to handle. Typically, these issues will result in a cascading effect across the system, and a proper management of the incidents depends heavily on a thorough analysis of this effect. Therefore, in this paper, we propose a method to automatically analyze the cascading effect of availability issues in online systems and extract the corresponding graph based issue representations incorporating both of the issue symptoms and affected service attributes. With the extracted representations, we train and utilize a graph neural networks based model to perform incident detection. Then, for the detected incident, we leverage the PageRank algorithm with a flexible transition matrix design to locate its root cause. We evaluate our approach using real-world data collected from the WeChat online service system, the largest instant message system in China. The results confirm the effectiveness of our approach. Moreover, our approach is successfully deployed in the company and eases the burden of operators in the face of a flood of issues and related alert signals.
@inproceedings{paperbot7271,
title = {Graph based Incident Extraction and Diagnosis in Large-Scale Online Systems},
author = {Zilong He and Pengfei Chen and Yu Luo and Qiuyu Yan and Hongyang Chen and Guangba Yu and Fangyuan Li},
booktitle = {ASE},
year = {2022},
doi = {10.1145/3551349.3556904}
}