一句话：提出意图引导的 RAG 聊天机器人 iKnow，通过意图检测、查询重写与缺失知识检测提升云运维问答准确率。
云运维文档庞大，工程师难以高效获取知识。检索增强生成（RAG）可检索相关知识并生成带引用答案，但面向运维问答（OpsQA）的可靠 RAG 聊天机器人仍具挑战。本文基于某大型云厂商的部署经验，对三个运维团队 2000 条真实查询进行实证研究，识别出五类 OpsQA 意图（如症状分析、术语解释）及其答案需求，并归纳六类失败根因——过半源于查询不完整、超范围或无效。为此提出 iKnow，集成意图检测、按意图定制的查询重写与缺失知识检测。内部评估中，回答准确率从 65.8% 提升到 81.3%，延迟仅小幅增加；iKnow 已在 CloudA 部署六个月，支持数千名工程师日常运维。
Managing complex cloud services requires standard operational documentation, but its sheer volume often hinders cloud engineers from efficient knowledge acquisition. Retrieval-Augmented Generation (RAG) can streamline this process by retrieving relevant knowledge and generating concise, referenced answers. However, deploying a reliable RAG-based chatbot for cloud operation remains a challenge. In this experience paper, we analyze the development and deployment of RAG-based chatbots for operational question answering (OpsQA) at a large-scale cloud vendor. Through an empirical study of 2,000 real-world queries across three operational teams, we identify five unique OpsQA intent types (e.g., symptom analysis and terminology explanation) and their corresponding requirements for a satisfactory answer, which differ from general software engineering queries. Our analysis further uncovers six root causes leading to chatbot failures—over half stem from query issues (i.e., incompleteness, out-of-scope, or invalid queries), while others are from retrieval or generation issues. To address these issues, we propose iKnow, an intent-guided RAG-based chatbot that integrates intent detection, query rewriting tailored to each intent, and missing knowledge detection to enhance answer quality. In internal evaluations, iKnow improves average answer accuracy from 65.8% to 81.3% with only a modest increase in latency. iKnow has been deployed for six months at CloudA, supporting thousands of cloud engineers in daily operations. We discuss lessons learned from real-world deployment, providing valuable insights for future research and practical implementations in similar domains.
DOI 原文 ·
@inproceedings{paperbot9269,
title = {iKnow: an Intent-Guided Chatbot for Cloud Operations with Retrieval-Augmented Generation},
author = {Junjie Huang and Yuedong Zhong and Guangba Yu and Zhihan Jiang and Minzhi Yan and Wenfei Luan and Tianyu Yang and Rui Ren and Michael R. Lyu},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00084}
}