一句话：提出 ScenarioBench 基准，审计 LLM 生成测试是否覆盖开发者真正关心的场景，并揭示其可能带来虚假安全感。
单元测试对软件开发很重要，但耗时繁琐，因此人们转向基于 LLM 的测试自动生成。问题在于，LLM 生成的测试通过只能说明代码相对于 LLM 的理解是正确的，而非相对于开发者的意图。为此，本文提出 ScenarioBench：由专家为 Defects4J 中 30 个方法整理测试场景，并使用五种先进 LLM 测试生成技术以及 Claude Code，在有无自然语言规约、代码正确或有缺陷等常见配置下生成测试。作者系统地将生成测试与专家场景比较，发现 LLM 方法在所有配置下都可能生成不正确、过度指定或不完整的测试套件，且配置对生成测试的影响小于预期。这表明通过的 LLM 生成测试套件可能让开发者误以为手工编写或 AI 生成的软件是正确的。
Unit testing is an important part of the software development process. But unit testing can be time-consuming and tedious, and so many people have turned to LLM-based approaches to automate the process. The problem is, a passing LLM-generated unit test suite provides evidence only that the code is correct with respect to the LLM's understanding, not the developer's. Our concern is that LLM-based approaches might come up with their own definition of "correct" behavior, separate from what the developer intended. Therefore, in this work, we investigate if LLM-generated tests verify the scenarios developers actually expect. Specifically, we propose a benchmark, ScenarioBench, of expert-curated test scenarios for a subset of 30 methods from the Defects4J dataset. We generate tests for these 30 Defects4J methods using five different state-of-the-art LLM-based techniques, as well as Claude Code, simulating different common configurations in software development: when a natural language specification is available, when it is not, when the code is correct, and when the code is buggy. We systematically compare the generated tests to the expert-curated test scenarios, and we find that LLM-based approaches can generate incorrect, overspecified, or incomplete test suites in all configurations, and that the configuration has less of an effect on the generated tests than expected. Our findings demonstrate how a passing LLM-generated test suite can provide developers with a false sense of security that their software, either manually written or AI-generated, is correct.