一句话：首个系统实证研究规约的「形式」与「粒度」两个维度如何影响 LLM 代码生成效果，发现模块级细粒度结构化伪代码最优。
论文关注 LLM 代码生成中的「vibe coding」实践：开发者用非结构化自然语言与 LLM 交互，仅凭整体执行输出验证实现，而自然语言的歧义与不一致性使其难以支撑复杂大规模软件系统的生成。为此，作者提出规约驱动开发范式下尚未被回答的问题——规约应以何种形式、何种粒度提供给 LLM。本文开展首个系统性实证研究，构造包含五个中等规模软件项目（数百至千余行代码、多语言、覆盖系统工具与游戏等领域）的数据集，在「形式」维度上设计自然语言、粗粒度伪代码、细粒度伪代码三种规约，在「粒度」维度上将细粒度伪代码划分为行级、函数级、模块级、项目级四种输入粒度；使用 GPT-5、DeepSeek-V3.1、Qwen-3.7 三个代表性模型生成代码，并以文本相似度指标（BLEU-4、CodeBLEU）、测试类指标（准确率）与人本指标（可用性、逻辑一致性）进行综合评估。结果显示：细粒度结构化伪代码显著提升生成的稳定性与质量；较粗粒度（项目级、模块级）规约通常带来更高质量，但模块级粒度在质量、稳定性与 token 效率之间取得最佳总体平衡，且对开发者最为友好。
Recent advances in large language models (LLMs) for code generation have given rise to vibe coding, a programming practice in which developers interact with LLMs through unstructured natural language and primarily validate implementations by inspecting overall execution outputs. However, the inherent ambiguity and inconsistency of natural language present substantial challenges for generating complex and large-scale software systems. To mitigate these limitations, an emerging AI programming paradigm—specification-driven development—advocates providing LLMs with precise and consistent specifications to guide code generation. Despite its promise, fundamental questions regarding the appropriate forms of specifications and the granularity at which they should be supplied to LLMs remain largely unexplored. To address this gap, we present the first empirical study that systematically investigates the impact of two orthogonal dimensions of specifications—form and granularity—on the effectiveness of LLM-based code generation. We construct a dataset comprising five medium-sized software projects, ranging from several hundred to over one thousand lines of code, implemented in multiple programming languages and spanning diverse application domains (e.g., system utilities and games). For each project, we design specifications in three forms—natural language, coarse-grained pseudocode, and fine-grained pseudocode—to capture the form dimension. To examine granularity, we further decompose the fine-grained pseudocode into four levels of input granularity: line-level, function-level, module-level, and project-level. Using this dataset, we evaluate code generated by three representative models, GPT-5, DeepSeek-V3.1, and Qwen-3.7. Evaluation is conducted using a comprehensive set of metrics, including text-similarity-based metrics (e.g., BLEU-4 and CodeBLEU), test-based metrics (e.g., accuracy), and human-centered metrics (e.g., usability and logical consistency). Our results demonstrate that using fine-grained, structured pseudocode specifications significantly enhances generation stability and quality. Regarding granularity, while coarser-grained specifications (project and module level) generally achieve higher code quality, organizing specifications at the module level provides the best overall balance between quality, stability, and token efficiency. Moreover, additional analyses indicate that this module-level granularity yields the most user-friendly outcome for human developers.
@inproceedings{paperbot12434,
title = {From Vibe to Spec: An Empirical Study to Demystify Specifications in Code Generation},
author = {Ruirui He and Hanqing Lu and Lezhi Ma and Kai Ye and Qiong Wu and Shangqing Liu and Lei Bu and Xuandong Li},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3837515}
}