一句话：用语义程序变体作为行为探针，自动生成回归测试以补齐仓库级修复基准的测试薄弱点。
SWE-bench 等基于测试的基准通过回归测试套件判定补丁是否正确，但弱测试套件会接受语义上错误却貌似合理的补丁，从而虚高自动修复 agent 的性能。本文提出 PROBE，一个测试增强框架：它对参考补丁进行语义修改得到程序变体，将仍能通过原始测试的“存活变体”视为行为探针，以揭示测试套件未约束的行为，并据此定向生成回归测试。生成的测试只有同时满足三个条件才被保留：在参考补丁上通过、在至少一个存活变体上失败、且在保持语义的程序变换下保持稳定。在 SWE-bench Verified 上，77% 的实例存在至少一个存活变体；PROBE 为 211 个实例生成 1,014 个经验证的测试，使补丁区域的行覆盖率和分支覆盖率分别提升 10.8 与 9.5 个百分点。用增强后的测试套件重新评估排名前 10 的修复 agent，解决率下降 4.2%–9.0%，表明许多此前被接受的补丁其实利用了基准测试的漏洞，而非真正满足预期修复语义。
Test-based benchmarks such as SWE-bench have become a standard basis for evaluating automated issue resolution agents, deeming a patch correct if it passes a provided regression test suite. In practice, weak test suites can admit plausible but semantically incorrect patches, inflating reported agent performance. We present PROBE, a test augmentation framework that uses semantically modified program variants as behavioral probes to identify and close gaps in benchmark test suites. Variants of the reference patch that survive the original tests reveal under-constrained behaviors, which then guide targeted regression test generation. Each generated test is retained only if it passes on the reference patch, fails on at least one surviving variant, and remains robust under behavior-preserving transformations. On SWE-bench Verified, 77% of instances admit at least one surviving variant. PROBE generates 1,014 validated tests across 211 instances, increasing patch-region line and branch coverage by 10.8 and 9.5 percentage points. Re-evaluating the top-10 repair agents with the augmented suites reduces resolved rates by 4.2%-9.0%, showing that many previously accepted patches exploit benchmark test gaps rather than fully satisfying the intended repair semantics. These findings demonstrate that benchmark evaluation is not solely a patch-generation problem but also a test-strength problem.
@inproceedings{paperbot12453,
title = {Probe to Generate: Program Variant-Guided Test Augmentation for Repository-Level Repair Benchmarks},
author = {Chenglin Li and Yisen Xu and Zehao Wang and Shin Hwei Tan and Tse-Hsun (Peter) Chen},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3837534}
}