一句话:AssertFlip 先生成通过测试再反转为失败测试,用 LLM 自动复现 Bug,在 SWT-Bench 上达到最佳效果。
问题:开源与工业环境中多数 bug 在报告时缺少可执行复现测试,使诊断和修复困难且耗时。方法:AssertFlip 利用 LLM 自动生成 Bug Reproducible Tests(BRT)。与直接生成失败测试不同,它先在含 bug 的行为上生成可通过的测试,再反转这些测试,使其在 bug 存在时失败;其假设是 LLM 更擅长编写通过测试。评估:在专为 BRT 构建的 SWT-Bench 上,AssertFlip 在 SWT-Bench-Verified 子集取得 43.6% 的 fail-to-pass 成功率,优于所有已知技术并位居 leaderboard 最佳。
Bug reproduction is critical in the software debugging and repair process, yet the majority of bugs in open-source and industrial settings lack executable tests to reproduce them at the time they are reported, making diagnosis and resolution more difficult and time-consuming. To address this challenge, we introduce AssertFlip, a novel technique for automatically generating Bug Reproducible Tests (BRTs) using large language models (LLMs). Unlike existing methods that attempt direct generation of failing tests, AssertFlip first generates passing tests on the buggy behaviour and then inverts these tests to fail when the bug is present. We hypothesize that LLMs are better at writing passing tests than ones that crash or fail on purpose. Our results show that AssertFlip outperforms all known techniques in the leaderboard of SWT-Bench, a benchmark curated for BRTs. Specifically, AssertFlip achieves a fail-to-pass success rate of 43.6% on the SWT-Bench-Verified subset.