一句话：Eagle 从企业运维文档自动合成覆盖日志、指标、追踪与配置的多粒度问答基准，用于评估运维大模型。
现代软件系统规模与复杂度的增长使智能可靠的 IT 运维愈发重要，但现有 AIOps 方案多为孤立的专用模型，难以处理多模态数据、维护成本高且缺乏透明性；运维大模型（OpsLLM）虽具备统一推理能力，其评估却受制于缺乏运维导向的分类体系、公开数据集稀缺、问题生成方法过于简单以及质量规范不足。本文提出 Eagle，一个面向 OpsLLM 的综合基准框架：它首先定义以运维为中心的分类体系，将大模型核心能力与端到端运维任务对齐；其次实现带多粒度质量控制的自动问题生成流水线，并通过人工标注验证其质量；最后提供可复现的评估套件与指标，支持场景驱动的推理评估。Eagle 已在华为内部部署，从企业产品文档中合成 4,845 条涵盖日志、指标、追踪与配置的领域问答对，支撑多轮 OpsLLM 评测并产出横向基准报告，用于模型选型与上线决策。离线研究中，Eagle 生成的测试套件在专家评分上较最先进基线提升 22%–49%，能更精确地评估异常检测、故障诊断与根因分析能力。框架与脱敏数据集已开源。
The rapid growth in scale and complexity of modern software systems has intensified the need for intelligent and reliable IT operations. While Artificial Intelligence for IT Operations (AIOps) addresses some challenges, existing solutions predominantly rely on isolated, task-specific models that struggle with interpreting multimodal data, incur high maintenance costs, and lack sufficient transparency. Operations Large Language Models (OpsLLMs) offer unified, knowledge-rich reasoning capabilities, yet their evaluation faces significant barriers, including the absence of Ops-centric evaluation taxonomies, limited availability of public datasets, simplistic question-generation methods, and inadequate quality standards for comprehensive operations tasks. We present Eagle, a comprehensive benchmarking framework tailored for evaluating OpsLLMs. Deployed inside Huawei, Eagle ingests enterprise product documentation and synthesizes 4,845 domain-grounded QA pairs across logs, metrics, traces, and configurations, which are paired with a standardized model evaluation system. This deployment supported multiple evaluations of OpsLLM and culminated in an internal horizontal benchmarking report that informed model selection and rollout decisions. Methodologically, Eagle (i) defines an operations-centric taxonomy aligning core LLM abilities with end-to-end operations tasks; (ii) implements an automated question-generation pipeline with multi-granular quality controls validated by human annotation; and (iii) provides reproducible evaluation suites and metrics for scenario-driven reasoning. In offline studies, Eagle-generated test suites improve expert-rated rubric scores by 22%–49% over state-of-the-art baselines, enabling more precise assessments of anomaly detection, fault diagnosis, and root-cause analysis abilities in OpsLLMs. To foster community adoption and reproducibility, we open-source the framework1 and a sanitized dataset2. By bridging general LLM evaluation and operations practice, Eagle delivers a deployable foundation for advancing large-model applications in AIOps.
@inproceedings{LiuPWXDFZJP26,
title = {Eagle: Leveraging Operations Documents for Comprehensive Benchmark Question Generation},
author = {Yuhe Liu and Changhua Pei and Hang Wang and Longlong Xu and Xiaogang Dong and Zhen Feng and Li Zheng and Kehang Ji and Dan Pei},
booktitle = {FSE},
year = {2026},
doi = {10.1145/3803437.3805214}
}