一句话：首次量化被测代码正确性对 LLM 测试生成的影响：错误代码令测试准确率、覆盖率与缺陷检测显著下降。
人们通常假设，与提示错误的被测代码相比，LLM 在提示正确的被测代码时更容易生成正确的测试用例，但这一效应的大小此前从未被测量。作者对 5 个开源与 6 个闭源语言模型开展大规模实证研究，覆盖 3 个广泛使用的基准数据集，以及来自两个真实数据集的 41 个仓库级真实样例。结果表明：相较于被测代码错误的情形，提示正确代码的 LLM 在测试准确率、代码覆盖率和缺陷检测上分别提升 57%、12% 与 24%；这些结论可从三个基准数据集推广到真实代码，为错误代码生成的测试其缺陷检测率要差 47%。作者还发现，提供自然语言代码描述可带来 +18% 准确率、+4% 覆盖率与 +34% 缺陷检测的提升。基于此，论文给出了面向实践的可操作结论与补救建议。
It is natural to suppose that a Large Language Model is more likely to generate correct test cases when prompted with correct code under test, compared to incorrect code under test. However, the size of this effect has never been previously measured, despite its obvious importance for both practicing software engineers and researchers. To answer the question, we conducted a comprehensive empirical study on 5 open source and 6 closed source language models, with 3 widely-used benchmark data sets together with 41 repo-level real-world examples from two different real-world data sets. Our results reveal that, when compared to incorrect code under test, LLMs prompted with correct code achieve improvements in test accuracy, code coverage, and bug detection of 57%, 12%, and 24% respectively. We further show that these scientific conclusions carry over from the three benchmark data sets to the real-world code, where tests generated for incorrect code experience a 47% worse bug detection rate. Finally, we report that improvements of +18% in accuracy, +4% coverage, and +34% in bug detection can be achieved by providing natural language code descriptions. These findings have actionable conclusions. For example, the 47% reduction in real-world bug detection is a clear concern. Fortunately, it is a concern for which our findings about the added value of descriptions offer an immediately actionable remedy.