一句话：对六个LLM在HumanEval上的代码生成错误做实证分析，建立错误分类并揭示其语义/句法特征与成因。
论文关注大语言模型生成代码时产生错误但缺乏系统理解的问题。作者在HumanEval上分析六个代表性LLM的代码生成错误，采用开放式编码与主题分析，从语义和句法两个维度归纳出错误分类，并考察错误位置、根因、与任务复杂度及测试通过率的相关性。结果显示，LLM常犯非平凡、多行且分散于不同位置的错误，根因多样；这些错误与任务复杂度和测试通过率存在关联，给定位和修复带来挑战。最后讨论了未来方向。
Large Language Models (LLMs) have demonstrated unprecedented capabilities in code generation. However, there remains a limited understanding of code generation errors that LLMs can produce. To bridge the gap, we conducted an in-depth analysis of code generation errors across six representative LLMs on the HumanEval dataset. Specifically, we first employed open coding and thematic analysis to distill a comprehensive taxonomy of code generation errors. We analyzed two dimensions of error characteristics-semantic characteristics and syntactic characteristics. Our analysis revealed that LLMs often made non-trivial, multi-line code generation errors in various locations and with various root causes. We further analyzed the correlation between these errors and task complexity as well as test pass rate. Our findings highlighted several challenges in locating and fixing code generation errors made by LLMs. In the end, we discussed several future directions to address these challenges.
DOI 原文 ·
@inproceedings{paperbot8815,
title = {Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models},
author = {Zhijie Wang and Zijie Zhou and Da Song and Yuheng Huang and Shengmai Chen and Lei Ma and Tianyi Zhang},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00180}
}