一句话：基于5个云服务四年数千起生产事故，定义并实证刻画「优质首条消息」（GFM），并提出 LLM 排序器 σ-rank 自动将其顶到前列。
现代云服务依赖 AI agent、规则化自动化与人类工程师在共享讨论线程中协同处置事故，值班工程师在分钟级时间压力下必须快速判断海量机器生成消息中哪些真正有用。论文在 5 个云服务、2022–2026 四年间的数千起生产事故上开展实证研究，定义「优质首条消息」（GFM）为线程中首条内容可证明影响值班工程师下一步调查动作的机器生成消息。三项发现：引入 AI agent 后线程长度中位数从 4 增至 30（7 倍），但噪声反而下降，AI 在 AI agent、规则自动化与人类三类主体中噪声率最低；AI 与自动化互补，承担诊断与监控等不同工作流角色；仅约五分之一事故含 GFM，其影响力随功能角色差异极大，而随主体类型差异很小——即「说了什么」比「谁说的」更重要。作者进一步提出 σ-rank：一个 prompt 中编码实证领域知识的 LLM 排序器，在约 20 条机器消息中 89% 的情况下将 GFM 排进前三，将工程师的噪声阅读负担降低 7 倍。
Modern cloud services increasingly rely on agentic software operations, where AI agents, rule-based automation (monitoring alerts, health checks, template runbooks), and human engineers coordinate incident response through shared discussion threads. As these systems mature, the on-call engineer faces a growing volume of machine-generated messages and must quickly identify which ones actually help, under minutes-level time pressure where every moment spent reading noise delays mitigation. We study this problem across thousands of production incidents from 5 cloud services over four years (2022 to 2026). We define a good first message (GFM): the first machine-generated message in an incident discussion thread whose specific content demonstrably influences the on-call engineer’s next investigation action. Three findings emerge. First, incident threads grew 7× longer (from a median of 4 to 30 messages) as AI agents were introduced, yet noise (messages lacking substantive technical content) decreased: among the three actor types, AI agents, rule-based automation, and humans, AI produces the lowest noise rate. Second, AI agents and automation are complementary, covering different workflow roles (e.g., diagnosis vs. monitoring) rather than duplicating each other. Third, only about one in five incidents contains a GFM; influence varies dramatically by functional role but far less by actor type: what a message says matters more than who posted it. We further show that GFMs can be surfaced automatically: σ-rank, an LLM ranker whose prompt encodes domain knowledge from the empirical study, places the GFM in the top 3 of ∼20 machine messages 89% of the time, reducing the engineer’s noise-reading burden by 7×. These results provide the first empirical evidence of how deployed AI output is actually consumed during live incidents and offer a practical mechanism to cut through information overload in time-critical incident response.
@inproceedings{paperbot12200,
title = {When Does AI Actually Help in Incident Response? Identifying Good First Messages in Cloud Service Incidents},
author = {Minghua Ma and Rujia Wang and Chetan Bansal and Saravan Rajmohan and Yingnong Dang and Hongyu Zhang},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3837473}
}