一句话：本文实证发现，过时注释会系统性误导 LLM 代码补全，使生成偏向历史实现，且影响受函数签名长度与任务复杂度调节。
论文研究 LLM 代码补全中自然语言注释作为上下文时的可靠性。尽管注释通常被认为有助于生成，但在演化代码库中可能过时或与实现不一致。作者开展实证研究，考察过时注释对补全的影响。结果发现，过时注释并非仅降低性能，而是系统性误导模型：生成偏向历史实现，在实现已变更区域出错，整体却仍看似合理，因为注释保留部分正确但不一致的信息。该效应随函数签名长度和任务复杂度变化，体现结构信号与自然语言引导的交互。研究挑战了丰富上下文必然有益的假设，强调注释是强但不可靠的信号，需要更稳健地处理不一致上下文。
Large language models (LLMs) are widely used for code completion, where natural-language comments serve as key contextual signals. While prior work shows that comments and natural-language descriptions can improve generation quality by providing high-level semantic guidance, their reliability in evolving codebases remains unclear. In practice, comments may be outdated or inconsistent with the current implementation, raising the question of whether such context may mislead generation rather than simply degrade it. In this paper, we present an empirical study of outdated comments in LLM-based code completion. We find that outdated comments tend to introduce a systematic misleading effect: instead of merely degrading performance, they can bias generated code toward historical implementations, producing outputs that remain globally plausible while failing in regions where the implementation has changed, as outdated comments often retain partially correct yet inconsistent information. Importantly, this effect is not uniform. Our study shows that the impact of outdated comments varies with factors such as the function signature length and task complexity, reflecting the interaction between structural signals and natural-language guidance. Overall, our findings challenge the common assumption that richer context is uniformly beneficial for code generation. They highlight that natural-language comments can act as strong but unreliable signals, and underscore the need for more robust handling of inconsistent contextual information in LLM-based code generation.
@inproceedings{paperbot12419,
title = {When Comments Mislead: An Empirical Study of Outdated Comments in LLM-Based Code Completion},
author = {Zhikun Xia and Yuan Huang and Xiangping Chen and Zibin Zheng},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834359}
}