一句话：提出 CHIME，通过 CFG 解析栈轨迹、蜕变测试和查询变换，缓解 ChatGPT 理解技术报告时的不准确，纠正率提升 30.3%。
针对 ChatGPT 在包含文本与技术术语的技术文本中易产生幻觉且缺乏系统评估的问题，作者首先调研 47 名软件工程师，并基于两个开源项目的缺陷报告构建了 412 个问答对的基准。实验发现，经 RAG 调整的 ChatGPT 回答正确率仅 36.4%，主要受限于难以理解代码片段（如栈轨迹）中的复杂技术内容，以及难以整合技术术语与文本所表示的上下文。为此提出 CHIME（ChatGPT Inaccuracy Mitigation Engine）：用上下文无关文法（CFG）解析栈轨迹，并通过蜕变测试与查询变换验证和修正 ChatGPT 的回答。在基准上，CHIME 相比原始 ChatGPT 回答的纠正率提升 30.3%；用户研究亦表明其改进回答更有用。
Hallucinations, the tendency to produce irrelevant/incorrect responses, are prevalent concerns in generative AIbased tools like ChatGPT. Although hallucinations in ChatGPT are studied for textual responses, it is unknown how ChatGPT hallucinates for technical texts that contain both textual and technical terms. We surveyed 47 software engineers and produced a benchmark of 412 Q&A pairs from the bug reports of two OSS projects. We find that a RAG-based ChatGPT (i.e., ChatGPT tuned with the benchmark issue reports) is 36.4 % correct when producing answers to the questions, due to two reasons 1) limitations to understand complex technical contents in code snippets like stack traces, and 2) limitations to integrate contexts denoted in the technical terms and texts. We present CHIME (ChatGPT Inaccuracy Mitigation Engine) whose underlying principle is that if we can preprocess the technical reports better and guide the query validation process in ChatGPT, we can address the observed limitations. CHIME uses context-free grammar (CFG) to parse stack traces in technical reports. CHIME then verifies and fixes ChatGPT responses by applying metamorphic testing and query transformation. In our benchmark, CHIME shows 30.3% more correction over ChatGPT responses. In a user study, we find that the improved responses with CHIME are considered more useful than those generated from ChatGPT without CHIME.
DOI 原文 ·
@inproceedings{paperbot8844,
title = {Chatgpt Inaccuracy Mitigation During Technical Report Understanding: Are we There Yet?},
author = {Salma Begum Tamanna and Gias Uddin and Song Wang and Lan Xia and Longyu Zhang},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00145}
}