一句话：首次实证研究LLM代码生成对歧义、矛盾与不完整任务描述的鲁棒性，揭示微小表述缺陷即可显著降低功能正确性。
该论文关注大语言模型在代码生成中面对不清晰任务描述时的鲁棒性问题。作者扩展 HumanEval 和 MBPP 基准，通过引导式变异策略系统注入歧义、不完整和自相矛盾等现实缺陷，构建更贴近非正式开发者指令的数据集。随后评估多种不同规模与架构的 LLM，按缺陷类别分析功能正确性与失败模式。结果表明，即使任务描述措辞存在微小缺陷也会导致性能显著下降，矛盾描述尤其引发大量逻辑错误；更大模型虽相对更稳健，但仍无法免疫。研究进一步分析语义错误模式，发现描述清晰度、模型行为与错误类型之间的关联，并强调需要提升模型对自然任务缺陷的鲁棒性，改进训练策略与更现实的评测基准。
Large Language Models (LLMs) have demonstrated impressive performance in code generation tasks under idealized conditions, where task descriptions are clear and precise. However, in practice, task descriptions frequently exhibit ambiguity, incompleteness, or internal contradictions. In this paper, we present the first empirical study examining the robustness of state-of-the-art code generation models when faced with such unclear task descriptions. We extend the HumanEval and MBPP benchmarks by systematically introducing realistic task descriptions flaws through guided mutation strategies, producing a dataset that mirrors the messiness of informal developer instructions. We evaluate multiple LLMs of varying sizes and architectures, analyzing their functional correctness and failure modes across task description categories. Our findings reveal that even minor imperfections in task description phrasing can cause significant performance degradation, with contradictory task descriptions resulting in numerous logical errors. Moreover, while larger models tend to be more resilient than smaller variants, they are not immune to the challenges posed by unclear requirements. We further analyze semantic error patterns and identify correlations between description clarity, model behavior, and error types. Our results underscore the critical need for developing LLMs that are not only powerful but also robust to the imperfections inherent in natural user tasks, highlighting important considerations for improving model training strategies, designing more realistic evaluation benchmarks, and ensuring reliable deployment in practical software development environments.
@inproceedings{abs-2507-20439,
title = {When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptions},
author = {Maya Larbi and Amal Akli and Mike Papadakis and Rihab Bouyousfi and Maxime Cordy and Federica Sarro and Yves Le Traon},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3773186}
}