一句话：提出 ESRO，通过融合告警因果图与故障报告知识图，为云服务故障推荐根因与修复方案。
现代云服务架构复杂，故障诊断困难。SRE 需结合告警、错误日志与历史经验定位根因，但历史故障报告中的半结构化经验难以系统利用，而告警数据又规模庞大。本文提出诊断服务 ESRO：利用告警构建因果图、利用故障报告构建知识图，并在训练阶段将二者融合为统一图；推理时基于检索机制在统一图中搜索，结合故障期间触发的告警及其对故障组的重要性，对候选根因与修复技术排序推荐。作者在大型 SaaS 企业约两年的云服务故障上评估，ROUGE 分数较 SOTA 基线平均提升 27%，并通过多个真实故障案例定性验证有效性。
Modern cloud services are prone to failures due to their complex architecture, making diagnosis a critical process. Site Reliability Engineers (SREs) spend hours leveraging multiple sources of data, including the alerts, error logs, and domain expertise through past experiences to locate the root cause(s). These experiences are documented as natural language text in outage reports for previous outages. However, utilizing the raw yet rich semi-structured information in the reports systematically is time-consuming. Structured information, on the other hand, such as alerts that are often used during fault diagnosis, is voluminous and requires expert knowledge to discern. Several strategies have been proposed to use each source of data separately for root cause analysis. In this work, we build a diagnostic service called ESRO that recommends root causes and remediation for failures by utilizing structured as well as semi-structured sources of data systematically. ESRO constructs a causal graph using alerts and a knowledge graph using outage reports, and merges them in a novel way to form a unified graph during training. A retrieval based mechanism is then used to search the unified graph and rank the likely root causes and remediation techniques based on the alerts fired during an outage at inference time. Not only the individual alerts, but their respective importance in predicting an outage group is taken into account during recommendation. We evaluated our model on several cloud service outages of a large SaaS enterprise over the course of ~2 years, and obtained an average improvement of 27% in rouge scores after comparing the likely root causes against the ground truth over state-of-the-art baselines. We further establish the effectiveness of ESRO through qualitative analysis on multiple real outage examples.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2309-07230,
title = {ESRO: Experience Assisted Service Reliability against Outages},
author = {Sarthak Chakraborty and Shubham Agarwal and Shaddy Garg and Abhimanyu Sethia and Udit Narayan Pandey and Videh Aggarwal and Shiv Saini},
booktitle = {ASE},
year = {2023},
doi = {10.1109/ase56229.2023.00131}
}