paperbot · PL 论文追踪

全部论文 待读 收藏

P2 PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

PAIChecker：发现并检查类 SWE-Bench 基准中的 PR-Issue 错位
ASE Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering2026匹配度 4/10 · 该论文关注类 SWE-Bench 基准中 PR 与 Issue 的错位问题，并提出多智能体系统 PAIChecker 进行检测。虽然涉及 LLM Agent 和基准质量，但未涉及静态分析、形式化验证、浮点程序分析或贝叶斯程序分析，也未将静态分析应用于 Agentic 程序，与用户核心研究方向匹配度较低。引用 0 inbox
Manyi Wang, Junjielong Xu, Pinjia He
机器学习与程序语言测试、调试与修复
crossref: 2026-10-09

AI 速览

一句话：提出多智能体系统 PAIChecker，检测类 SWE-Bench 基准中 PR 与 Issue 的错位，提升基准可靠性。

SWE-bench 类基准通常从 PR 描述中提取 issue 引用进行配对，将 issue 描述作为问题陈述、PR 补丁作为测试预言，但大型仓库的复杂性常导致 PR-Issue 错位。本文系统研究 SWE-bench Verified，发现 13.6% 实例存在错位，涵盖五类模式、十一种细粒度场景。为支持可靠且可扩展的基准构建，作者提出多智能体系统 PAIChecker，采用三阶段设计：特定模式识别、跨智能体标签合成与代码级验证，实现更准确、可泛化且逐步验证的检测。在 SWE-Gym 与 SWE-bench Multilingual 上，以四个 LLM 为后端，PAIChecker 均取得最佳性能，二分类准确率最高分别达 92.12% 和 91.67%。

问题
类 SWE-Bench 基准通过 PR 描述中的 issue 引用建立 PR-Issue 配对；由于大型仓库开发维护复杂，这种配对常出现错位，影响基于此类基准的 LLM 评测可靠性。
方法
系统分析 SWE-bench Verified 中的错位模式，并设计三阶段多智能体系统 PAIChecker，结合模式识别、跨智能体标签合成与代码级验证来检测 PR-Issue 错位。
评估
在 SWE-Gym 与 SWE-bench Multilingual 上，以四个 LLM 后端评估，PAIChecker 的二分类准确率最高分别达 92.12% 和 91.67%。
原文摘要（Abstract）

SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired with its linked issue by extracting issue references from the PR description; the issue description is used as the problem statement, and the PR patch serves as the test oracle. However, due to the inherent complexity of developing and maintaining large repositories, such PR-Issue pairings are often misaligned in practice. In this work, we systematically study SWE-bench Verified instances, finding that 13.6% exhibit misalignment across five patterns in eleven fine-grained scenarios. To enable reliable and scalable construction of those benchmarks in the future, we propose PAIChecker, a multi-agent system for checking PR-Issue misalignment in SWE-bench-like benchmarks. Specifically, PAIChecker adopts a three-phase design that combines specific pattern identification, cross-agent label synthesis, and code-level validation, thereby enabling more accurate, generalizable, and progressively verified detection. Experiments on SWE-Gym and SWE-bench Multilingual show that PAIchecker achieves the best performance across all four LLM backbones, reaching up to 92.12% and 91.67% binary accuracy, respectively.

链接与引用

DOI 原文 · arXiv · DBLP

BibTeX 
@inproceedings{abs-2607-28587,
  title = {PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks},
  author = {Manyi Wang and Junjielong Xu and Pinjia He},
  booktitle = {ASE},
  year = {2026},
  doi = {10.1145/3832783.3837557}
}