一句话：眼动追踪与访谈显示，LLM 标注使评审者注视更久但审查彻底度不变，并改变其评审策略。
现代软件开发常用 LLM 生成代码，但 LLM 易出错和幻觉，需仔细审查；然而开发者信任与实际审查行为是否一致尚不清楚。本文用 Wizard-of-Oz 实验，让工程师评审被显式标注为 LLM 生成的代码，结合眼动追踪与退出访谈，并用贝叶斯和定性分析。结果显示，审查彻底程度未变，但参与者对 LLM 标注代码注视更久，说明标签影响注意力；他们还会按逻辑正确性等标准评估，或用提示词引导评审。研究为 LLM 工具标注设计与提示词作为软件制品提供启示，并揭示评审者意图与实际行为的差距，建议公司重审 AI 政策。
Modern software development increasingly involves the use of large language models (LLMs) to generate code. Despite their rapid advancement, LLMs remain prone to errors and hallucinations, emphasizing the importance of careful code inspection. However, in practice, developers' trust in LLM-generated code and their willingness to review it thoroughly may not align with these recommendations, which is a gap that remains largely unexplored. In this study, we conduct a Wizard-of-Oz experiment to examine how software engineers behave when code is explicitly labelled as LLM-generated during a code review task. We collect both behavioral data and participant feedback through eye-tracking and exit interviews. Combining Bayesian data analysis with qualitative analysis, we found that while the thoroughness of code review did not change for participants, they spent more time fixating on LLM-labelled code, indicating that the label itself influences attention. Practitioners also adapted their review strategy for LLM-labelled code by assessing the code based on specific criteria (e.g., logical correctness), or using the prompt to guide their review. These findings inform LLM-based tool design on labelling while incorporating the prompt as a software artifact. Our study reveals a gap between reviewers' intentions and actual reviewing behaviour, highlighting the need for software companies to revisit their AI policies (particularly regarding LLM-assisted development) to better support developers in reviewing LLM-generated code.
@inproceedings{abs-2606-26505,
title = {Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code},
author = {Ranim Khojah and Francisco Gomes de Oliveira Neto and Mazen Mohamad and Julian Frattini and Philipp Leitner},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834423}
}