一句话：实证表明在提示中加入测试用例能显著提升LLM函数级代码生成的成功率，验证TDD范式在AI辅助编程中的价值。
该文探讨能否以及如何将测试驱动开发（TDD）融入AI辅助的代码生成流程。传统TDD要求先写测试再写实现，从而保证代码与需求一致；作者假设在给LLM提供问题描述之外再提供测试用例，可以提升生成代码的正确性。实验在MBPP与HumanEval两个函数级代码生成基准上进行，对比仅给问题描述与同时给出测试用例两种提示方式，并使用GPT-4与Llama 3等模型。结果一致表明，包含测试用例的提示在解决编程任务上取得更高的成功率，说明测试可作为有效的规约与验证信号。作者据此主张TDD是帮助确保LLM生成代码真正满足需求的一种有前景的范式。
Recent Large Language Models (LLMs) have demonstrated significant capabilities in generating code snippets directly from problem statements. This increasingly automated process mirrors traditional human-led software development, where code is often written in response to a requirement. Historically, Test-Driven Development (TDD) has proven its merit, requiring developers to write tests before the functional code, ensuring alignment with the initial problem statements. Applying TDD principles to LLM-based code generation offers one distinct benefit: it enables developers to verify the correctness of generated code against predefined tests. This paper investigates if and how TDD can be incorporated into AI-assisted code-generation processes. We experimentally evaluate our hypothesis that providing LLMs like GPT-4 and Llama 3 with tests in addition to the problem statements enhances code generation outcomes. We experimented with established function-level code generation benchmarks such as MBPP and HumanEval. Our results consistently demonstrate that including test cases leads to higher success in solving programming challenges. We assert that TDD is a promising paradigm for helping ensure that the code generated by LLMs effectively captures the requirements.
DOI 原文 ·
@inproceedings{paperbot8309,
title = {Test-Driven Development and LLM-based Code Generation},
author = {Noble Saji Mathews and Meiyappan Nagappan},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695527}
}