paperbot · PL 论文追踪

全部论文 待读 收藏

Measuring the Cost of Prompt Structure in LLM Code Generation

度量 LLM 代码生成中提示结构的代价
ASE Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering2026引用 0 inbox
Yuetian Mao
crossref: 2026-10-09

AI 速览

一句话:在 DS-1000 与 SWE-bench Lite 上比较三级提示结构,发现结构只提升可执行性与格式合规,未必提升任务正确性。

结构化提示常能提升通用 LLM 推理,但其能否迁移到代码生成仍不清楚,而编码工具又在用户请求外添加模板层。本文在 DS-1000 和 SWE-bench Lite 上比较三级自动施加的提示结构:自由形式指令、静态组件模板,以及提取并强制显式输出契约的流水线。结果显示:中等结构使代码更易执行,但并未解决更多任务;严格契约强制提升格式合规性,却显著降低功能正确性,且效果依赖底层模型。作者据此认为,对代码生成而言提示结构是调参旋钮而非最佳实践,输出契约更适合放在验证而非生成阶段。

问题
结构化提示在通用推理中常有效,但它是否能迁移到代码生成尚缺乏系统研究,而实际编码工具又不断在用户请求外加入模板层。
方法
在 DS-1000 与 SWE-bench Lite 上,对三级自动施加的提示结构进行对照实验:自由形式指令、静态组件模板,以及提取并强制显式输出契约的流水线。
评估
以代码可执行性、任务解决率和输出格式合规性为指标,比较不同提示结构及不同底层模型下的表现。
原文摘要(Abstract)

Structured prompting routinely improves general LLM reasoning, but whether it transfers to code generation remains underexplored even as coding tools add template layers around user requests. We compare three levels of automatically imposed prompt structure, from free-form instructions over a static component template to a pipeline that extracts and enforces an explicit output contract, on DS-1000 and SWE-bench Lite. We find that moderate structure makes code more executable but does not resolve more tasks, strict contract enforcement improves format compliance while substantially reducing functional correctness, and the effect depends on the underlying model. For code generation, prompt structure is a tuning knob rather than a best practice, and output contracts belong in validation rather than generation.

链接与引用

DOI 原文 · PDF(开放获取) ·

BibTeX
@inproceedings{paperbot12311,
  title = {Measuring the Cost of Prompt Structure in LLM Code Generation},
  author = {Yuetian Mao},
  booktitle = {ASE},
  year = {2026},
  doi = {10.1145/3832783.3844577}
}