一句话：提出 Oasis：聚合相关 incident 自动评估故障影响范围，并用微调大语言模型生成人类可读的故障总结。
云应用一旦遭遇故障，用户会经历响应变慢、连接异常乃至服务中断，带来严重业务损失。云故障通常由多个并发事件与根因交织而成，理解其上下文是缓解与恢复的关键第一步，但当前实践中依赖具备深厚领域知识的 on-call 工程师手工评估与总结，既耗时又费力。本文首先对 Microsoft 内部 on-call 工程师处理云故障的现状开展大规模实证研究，随后提出并验证了 Oasis 方法：它通过多种技术聚合相关 incident 来评估故障的影响范围，并借助 GPT-3.x 等微调后的大语言模型生成人类可读的总结。其中影响范围评估组件已在 Microsoft 部署三年并被广泛采用，总结组件则为近期引入。作者在 18 个真实云系统上进行了实证评估，并邀请故障负责人参与人工评估，结果表明 Oasis 能高效且有效地总结故障，促使 Microsoft 部署了首个原型，目前正由部分事故响应团队试验性采用。
Cloud systems have become increasingly popular in recent years due to their flexibility and scalability. Each time cloud computing applications and services hosted on the cloud are affected by a cloud outage, users can experience slow response times, connection issues or total service disruption, resulting in a significant negative business impact. Outages are usually comprised of several concurring events/source causes, and therefore understanding the context of outages is a very challenging yet crucial first step toward mitigating and resolving outages. In current practice, on-call engineers with in-depth domain knowledge, have to manually assess and summarize outages when they happen, which is time-consuming and labor-intensive. In this paper, we first present a large-scale empirical study investigating the way on-call engineers currently deal with cloud outages at Microsoft, and then present and empirically validate a novel approach (dubbed Oasis) to help the engineers in this task. Oasis is able to automatically assess the impact scope of outages as well as to produce human-readable summarization. Specifically, Oasis first assesses the impact scope of an outage by aggregating relevant incidents via multiple techniques. Then, it generates a human-readable summary by leveraging fine-tuned large language models like GPT-3.x. The impact assessment component of Oasis was introduced in Microsoft over three years ago, and it is now widely adopted, while the outage summarization component has been recently introduced, and in this article we present the results of an empirical evaluation we carried out on 18 real-world cloud systems as well as a human-based evaluation with outage owners. The results obtained show that Oasis can effectively and efficiently summarize outages, and lead Microsoft to deploy its first prototype which is currently under experimental adoption by some of the incident teams.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{JinZML0LL0Z0HSD23,
title = {Assess and Summarize: Improve Outage Understanding with Large Language Models},
author = {Pengxiang Jin and Shenglin Zhang and Minghua Ma and Haozhe Li and Yu Kang and Liqun Li and Yudong Liu and Bo Qiao and Chaoyun Zhang and Pu Zhao and Shilin He and Federica Sarro and Yingnong Dang and Saravan Rajmohan and Qingwei Lin and Dongmei Zhang},
booktitle = {FSE},
year = {2023},
doi = {10.1145/3611643.3613891}
}