一句话：通过 HumanEval 上 GPT-4o 的实证分析，揭示错误或错误实现的假设是代码生成失败的重要诱因。
本文关注 LLM 代码生成中错误程序频发的问题，尤其是需求误解与逻辑不一致是否源于模型隐含假设。作者在 HumanEval 数据集上提示 GPT-4o 生成 Python 解，并要求其列出最重要的 3 个假设；随后用数据集测试用例验证输出。在 164 个解中发现 17 个错误程序，并分析这 17 个失败及对应的 51 个假设。结果显示，约 53% 的失败与 GPT 自身提出但错误或错误实现的假设直接相关，总计 71% 的代码生成失败与假设的错误提出或实现有关，表明显式化并校验模型假设有助于理解和减少 LLM 代码错误。
Large Language Models (LLMs) like ChatGPT are increasingly integrated into software development environments due to their strong performance in code generation. However, they often struggle with complex logic, security vulnerabilities, and code quality issues. These problems frequently originate from misunderstandings of problem requirements and logical inconsistencies, which can lead to faulty or vulnerable software. In this study, we conduct an initial empirical analysis to investigate the causes of erroneous code generated by the state-of-the-art LLM model GPT-4o. Using the HumanEval dataset, we prompt GPT-4o to generate Python solutions and list its 3 most important assumptions. We validate these outputs against the provided test cases in dataset and identify 17 defective programs out of 164 total solutions. By analyzing the 17 failures and 51 assumptions made on these tasks, we find that about 53% the failures are directly related to wrong or erroneously implemented assumptions raised by the GPT model itself, and totally 71% of code generation failures are related to erroneously made or implemented assumptions.
DOI 原文 ·
@inproceedings{paperbot9482,
title = {How Does ChatGPT Make Assumptions When Creating Erroneous Programs?},
author = {Sadia Jahan and Xiaoyin Wang},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00321}
}