一句话：提出覆盖多子领域的 IT 运维评测基准 OpsEval，含 7334 道选择题、1736 道问答题及运维专用问答评估方法。
云计算、5G 与金融平台等 IT 系统日益复杂，保障其稳定可靠成为关键挑战；LLM 在故障根因分析、运维脚本生成、告警摘要等 AIOps 任务上展现潜力，但运维知识随系统而异、含大量私有子领域术语且对提示工程敏感，现有 NLP 基准与 BLEU/ROUGE 指标难以指导 OpsLLM 的选择与评估。本文提出 OpsEval 基准套件，包含 7334 道选择题、1736 道问答题的运维数据集，配套评测基准与专为运维问答设计的评估方法，并公开领域专家撰写的 20% 数据。作者在 self-consistency、chain-of-thought、in-context learning 等设置下测试 24 个以上 LLM，总结出应用 LLM 于运维的发现；所提评估方法与人类专家相关系数为 0.9185，较 BLEU、ROUGE 分别提升 0.4471 与 1.366。
In recent decades, the field of software engineering has driven the rapid evolution of Information Technology (IT) systems, including advances in cloud computing, 5G networks, and financial information platforms. Ensuring the stability, reliability, and robustness of these complex IT systems has emerged as a critical challenge. Large language models (LLMs) that have exhibited remarkable capabilities in NLP-related tasks are showing great potential in AIOps, such as root cause analysis of failures, generation of operations and maintenance scripts, and summarizing of alert information. Unlike knowledge in general corpora, knowledge of Ops varies with the different IT systems, encompassing various private sub-domain knowledge, sensitive to prompt engineering due to various sub-domains, and containing numerous terminologies. Existing NLP-related benchmarks can not guide the selection of suitable LLMs for Ops (OpsLLM), and current metrics (e.g., BLEU, ROUGE) can not adequately reflect the question-answering (QA) effectiveness in the Ops domain. We propose a comprehensive benchmark suite, OpsEval, including an Ops-oriented evaluation dataset, an Ops evaluation benchmark, and a specially designed Ops QA evaluation method. Our dataset contains 7,334 multiple-choice questions and 1,736 QA questions. We have carefully selected and released 20% of the dataset written by domain experts in various sub-domains to assist current researchers in preliminary evaluations of OpsLLMs1. We test over 24 latest LLMs under various settings such as self-consistency, chain-of-thought, and in-context learning, revealing findings when applying LLMs to Ops. We also propose an evaluation method for QA in Ops, which has a coefficient of 0.9185 with human experts and is improved by 0.4471 and 1.366 compared to BLEU and ROUGE, respectively. Over the past one year, our dataset and leaderboard have been continuously updated.
@inproceedings{paperbot9098,
title = {OpsEval: A Comprehensive Benchmark Suite for Evaluating Large Language Models’ Capability in IT Operations Domain},
author = {Yuhe Liu and Changhua Pei and Longlong Xu and Bohan Chen and Mingze Sun and Zhirui Zhang and Yongqian Sun and Shenglin Zhang and Kun Wang and Haiming Zhang and Jianhui Li and Gaogang Xie and Xidao Wen and Xiaohui Nie and Minghua Ma and Dan Pei},
booktitle = {FSE},
year = {2025},
doi = {10.1145/3696630.3728572}
}