paperbot · PL 论文追踪

全部论文 待读 收藏

P2 HalluJudge: A Reference-Free Hallucination Detection for Context Misalignment in Code Review Automation

HalluJudge：面向代码评审自动化中上下文错位的无参考幻觉检测
FSE Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering2026匹配度 4/10 · 论文关注 LLM 生成代码评审评论的幻觉检测，属于 LLM 输出可信度评估，与用户关注的 Coding Agent/代码相关场景有间接联系，且具备工业落地（Atlassian）与问题驱动特征；但其方法基于 LLM 推理策略（如 Tree-of-Thoughts），并非静态分析、浮点或贝叶斯程序分析，也未把 LLM/Agent 当作改进静态分析的工具，核心仍偏向 LLM 自身输出质量评估，与用户研究核心交集有限。引用 7 inbox
Kla Tantithamthavorn, Hong Yi Lin, Patanamon Thongtanunam, Wachiraphan Charoenwet, Minwoo Jeong, Ming Wu
机器学习与程序语言编程环境与开发者工具
crossref: 2026-09-12dblp: 2026-09-12arxiv: 2026-09-12

AI 速览

一句话：提出无需参考的幻觉检测方法 HalluJudge，利用上下文对齐评估 LLM 代码评审评论是否有据，兼顾效果与成本。

LLM 在代码评审评论生成中表现突出，但会产生与代码事实不符的幻觉，阻碍其在评审流程中的采用。本文研究无参考条件下检测 LLM 生成代码评审评论幻觉的方法，设计 HalluJudge，基于生成评论与代码上下文的对齐程度评估其有据性，涵盖从直接评估到结构化多分支推理（如 Tree-of-Thoughts）的四种策略。作者在 Atlassian 企业级软件项目上系统评估这些策略的效果与成本效率，并分析 HalluJudge 判断与真实生产环境中开发者对 LLM 生成评论偏好的一致性。结果显示，HalluJudge 达到 0.85 的 F1 分数，平均成本仅 0.009 美元；平均 67% 的判断与开发者偏好一致。这表明其可作为实用防护手段，降低开发者接触幻觉评论的风险，增强对 AI 辅助代码评审的信任。

问题
LLM 生成的代码评审评论可能缺乏代码依据（幻觉），且在没有参考评论时难以自动检测，影响代码评审自动化的可信采用。
方法
提出 HalluJudge，基于生成评论与代码上下文的上下文对齐进行无参考幻觉评估，包含直接评估与结构化多分支推理（如 Tree-of-Thoughts）等四种策略。
评估
在 Atlassian 企业级软件项目上评估效果与成本，并分析其判断与真实生产环境中开发者偏好的一致性；F1 为 0.85，平均成本 0.009 美元，67% 判断与开发者偏好一致。
原文摘要（Abstract）

Large Language models (LLMs) have shown strong capabilities in code review automation, such as review comment generation, yet they suffer from hallucinations—where the generated review comments are ungrounded in the actual code—poses a significant challenge to the adoption of LLMs in code review workflows. To address this, we explore effective and scalable methods for a hallucination detection in LLM-generated code review comments without the reference. In this work, we design HalluJudge that aims to assess the grounding of generated review comments based on the context alignment. HalluJudge includes four key strategies ranging from direct assessment to structured multi-branch reasoning (e.g., Tree-of-Thoughts). We conduct a comprehensive evaluation of these assessment strategies across Atlassian's enterprise-scale software projects to examine the effectiveness and cost-efficiency of HalluJudge. Furthermore, we analyze the alignment between HalluJudge's judgment and developer preference of the actual LLM-generated code review comments in the real-world production. Our results show that the hallucination assessment in HalluJudge is cost-effective with an F1 score of 0.85 and an average cost of $0.009. On average, 67% of the HalluJudge assessments are aligned with the developer preference of the actual LLM-generated review comments in the online production. Our results suggest that HalluJudge can serve as a practical safeguard to reduce developers' exposure to hallucinated comments, fostering trust in AI-assisted code reviews.

链接与引用

DOI 原文 · arXiv · PDF（开放获取） · DBLP

BibTeX 
@inproceedings{abs-2601-19072,
  title = {HalluJudge: A Reference-Free Hallucination Detection for Context Misalignment in Code Review Automation},
  author = {Kla Tantithamthavorn and Hong Yi Lin and Patanamon Thongtanunam and Wachiraphan Charoenwet and Minwoo Jeong and Ming Wu},
  booktitle = {FSE},
  year = {2026},
  doi = {10.1145/3803437.3805236}
}