一句话：提出基于事件图的根因分析系统 Groot，在 eBay 生产环境支持 5000 个服务并实现高准确率。
针对大规模分布式系统与微服务架构下根因分析面临的操作复杂、系统规模大和监控困难等挑战，本文提出 Groot。Groot 基于事件构建实时因果图，事件汇总系统中的指标、日志和活动，并支持用户自定义事件与领域规则以融入 SRE 工程师的领域知识。目前 Groot 已在 eBay 生产环境中支持 5000 个服务，并被 SRE 团队实际使用。作者在 15 个月内收集了 952 个真实生产事故的标注根因数据集进行评估，结果显示 Groot 达到 95% 的 top-3 准确率和 78% 的 top-1 准确率。用户调研表明 Groot 有用且易用，论文还分享了在生产环境中部署和应用 Groot 的经验教训。
For large-scale distributed systems, it is crucial to efficiently diagnose the root causes of incidents to maintain high system availability. The recent development of microservice architecture brings three major challenges (i.e., complexities of operation, system scale, and monitoring) to root cause analysis (RCA) in industrial settings. To tackle these challenges, in this paper, we present Groot, an event-graph-based approach for RCA. Groot constructs a real-time causality graph based on events that summarize various types of metrics, logs, and activities in the system under analysis. Moreover, to incorporate domain knowledge from site reliability engineering (SRE) engineers, Groot can be customized with user-defined events and domain-specific rules. Currently, Groot supports RCA among 5,000 real production services and is actively used by the SRE teams in eBay, a global e-commerce system serving more than 159 million active buyers per year. Over 15 months, we collect a data set containing labeled root causes of 952 real production incidents for evaluation. The evaluation results show that Groot is able to achieve 95% top-3 accuracy and 78% top-1 accuracy. To share our experience in deploying and adopting RCA in industrial settings, we conduct a survey to show that users of Groot find it helpful and easy to use. We also share the lessons learned from deploying and adopting Groot to solve RCA problems in production environments.
DOI 原文 ·
@inproceedings{paperbot6602,
title = {Groot: An Event-graph-based Approach for Root Cause Analysis in Industrial Settings},
author = {Hanzhang Wang and Zhengkai Wu and Huai Jiang and Yichao Huang and Jiamu Wang and Selcuk Kopru and Tao Xie},
booktitle = {ASE},
year = {2021},
doi = {10.1109/ase51524.2021.9678708}
}