一句话：提出 Orchid 基准，首次系统实证研究需求歧义对 LLM 代码生成的影响，发现歧义普遍降低模型性能。
真实软件开发中的需求歧义源于自然语言的不精确与干系人理解差异，而现有代码生成基准普遍假设需求清晰无歧义，LLM 在歧义需求下的可靠性缺乏实证认识。本文提出 Orchid，一个面向歧义需求的函数级代码生成基准，包含 1304 个任务，覆盖词汇、句法、语义与模糊四类歧义，并据此开展首个系统性实证研究。结果表明：歧义一致地降低所有被评测 LLM 的性能，且能力越强的模型受损越明显；模型还常对同一歧义需求给出功能上相互分歧的实现，且无法自主识别或消解歧义。这揭示了清晰与歧义需求之间显著的性能差距，表明下一代自动化软件工程工具亟需歧义感知技术。
Software requirement ambiguity is ubiquitous in real-world development, stemming from the inherent imprecision of natural language and the varying interpretations of stakeholders. While Large Language Models (LLMs) have demonstrated impressive capabilities in generating code from precise specifications, such ambiguity poses a significant obstacle to reliable automated code generation. Existing benchmarks typically assume clear and unambiguous requirements, leaving an empirical gap in understanding how LLMs behave when faced with the inherent uncertainty of real-world software requirements. In this paper, we introduce Orchid, a code generation benchmark specifically designed with ambiguous requirements. It comprises 1,304 function-level tasks covering four distinct types of ambiguity: lexical, syntactic, semantic, and vagueness. Leveraging this dataset, we conduct the first systematic empirical study to evaluate the impact of requirement ambiguity on LLM-based code generation. Our results demonstrate that ambiguity consistently degrades the performance of all evaluated LLMs, with the most pronounced negative effects observed in highly advanced models. Furthermore, we observe that LLMs frequently produce functionally divergent implementations for the same ambiguous requirement and lack the capability to identify or resolve such ambiguity autonomously. These findings reveal a significant performance gap between clear and ambiguous requirements, underscoring the urgent need for ambiguity-aware techniques in the next generation of automated software engineering tools. The Orchid benchmark is publicly available at https://huggingface.co/datasets/SII-YDD/Orchid.
@inproceedings{paperbot12319,
title = {Clarity Is Not Assumed: Understanding LLM-Based Code Generation under Ambiguous Requirements},
author = {Di Yang and Xinou Xie and Xiuwen Yang and Ming Hu and Yihao Huang and Yueling Zhang and Weikai Miao and Ting Su and Chengcheng Wan and Geguang Pu},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834403}
}