一句话：首次大规模评估 GPT-3.x 等大语言模型在云生产事故根因定位与缓解建议中的效果。
云服务事故管理流程复杂，根因定位与缓解高度依赖值班工程师的领域知识和人工投入。本文在微软开展首个大规模研究，利用超过 40,000 个生产事故，评估 GPT-3.x（GPT-3.0 与 GPT-3.5）等先进大语言模型辅助工程师进行根因分析和缓解步骤推荐的有效性。作者在零样本、微调与多任务设置下比较多个大语言模型，并采用语义指标和词汇指标进行评价。最后，通过真实事故负责人的人工评估，验证了使用人工智能解决云事故的功效与未来潜力。
Incident management for cloud services is a complex process involving several steps and has a huge impact on both service health and developer productivity. On-call engineers require significant amount of domain knowledge and manual effort for root causing and mitigation of production incidents. Recent advances in artificial intelligence has resulted in state-of-the-art large language models like GPT-3.x (both GPT-3.0 and GPT-3.5), which have been used to solve a variety of problems ranging from question answering to text summarization. In this work, we do the first large-scale study to evaluate the effectiveness of these models for helping engineers root cause and mitigate production incidents. We do a rigorous study at Microsoft, on more than 40,000 incidents and compare several large language models in zero-shot, fine-tuned and multi-task setting using semantic and lexical metrics. Lastly, our human evaluation with actual incident owners show the efficacy and future potential of using artificial intelligence for resolving cloud incidents.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{AhmedGBZZR23,
title = {Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models},
author = {Toufique Ahmed and Supriyo Ghosh and Chetan Bansal and Thomas Zimmermann and Xuchao Zhang and Saravan Rajmohan},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00149}
}