一句话：通过差分补丁测试揭示 SWE-bench 验证机制会高估自动问题解决工具的解决率，并导致错误补丁被判正确。
论文关注 SWE-bench 及 SWE-bench Verified 用测试验证自动生成补丁的可靠性问题：测试不穷尽时，补丁可能通过测试却不符合开发者意图。作者对 CodeStory、LearnByInteract 和 OpenHands 在 SWE-bench Verified 上生成的补丁进行实证研究，提出差分补丁测试技术 PatchDiff，自动暴露两个补丁之间的行为差异，并与人工真值补丁比较。结果发现，7.8% 的补丁通过 SWE-bench 验证却未通过开发者测试套件；29.6% 的合理补丁与真值补丁行为不同，其中 46.8% 源于相似但发散的实现，27.3% 是生成补丁改变了更多行为，人工检查表明 28.6% 的行为发散补丁确定错误；综合导致报告解决率虚高 6.4 个百分点。
Automated issue solving aims to resolve real-world issues in software repositories. The most popular benchmarks for automated issue solving are SWE-bench and its human-filtered subset SWE-bench Verified, which are widely used to evaluate foundation models and software engineering agents. These benchmarks leverage testing to validate generated patches. However, because testing is rarely exhaustive, a patch may pass the tests but nevertheless fail to match the developers’ expectations. Unfortunately, it is currently unclear to what extent evaluations performed with SWE-bench suffer from such plausible but incorrect patches. This paper presents an in-depth empirical study of the correctness of plausible patches generated by three state-of-the-art issue-solving tools (CodeStory, LearnByInteract, and OpenHands) evaluated on SWE-bench Verified. We extensively test and inspect generated patches, and compare them against human-written ground truth patches. The core of our methodology is a novel technique for differential patch testing, called PatchDiff, which automatically exposes behavioral discrepancies between two patches. Our findings reveal critical weaknesses in SWE-bench’s patch validation mechanism, which causes 7.8% of all patches to count as “correct” while failing the developer-written test suite. Moreover, our novel automated technique reveals that even more (29.6%) plausible patches induce different behavior than the ground truth patches. These behavioral differences are often due to similar, but divergent implementations (46.8%) and due to generated patches that adapt more behavior than the ground truth patches (27.3%). Our manual inspection shows that 28.6% of behaviorally divergent patches are certainly incorrect. Combined, the different weaknesses lead to an inflation of reported resolution rates by 6.4 absolute percent points. Our findings are a call to arms for more robust and reliable evaluation of issue-solving tools. We envision our automated differential patch testing technique to be useful for this purpose.