一句话：提出 RepoProbe 基准，用 GitHub Discussions 开放式问答与检查表验证协议评测 LLM 的仓库架构理解，并量化 edit bias。
随着大语言模型（LLM）被用于软件工程，研究重心已从函数级代码生成转向仓库级辅助。然而现有基准多基于 GitHub Issues 中的缺陷报告，模型可借助错误日志上的模式匹配绕过真正的理解，从而低估了 Edit Bias——即模型在理解既有仓库架构之前就过早提出代码修改的倾向；同时 LLM-as-a-Judge 的标量打分方差大、可解释性差。本文提出 RepoProbe，一个基于 GitHub Discussions 的开放式问答基准，聚焦架构性提问而非缺陷报告；并提出基于检查表的验证协议，将答案分解为原子化、可验证的事实，以客观验证取代主观评分。对当前最优 LLM 的评测显示，答案表述清晰度与有证据支撑的技术正确性之间存在持续差距，并定量证实了 edit bias 的普遍性；该验证协议相比传统标量打分显著提升了评测可靠性。
The integration of Large Language Models (LLMs) into software engineering has shifted the focus from function-level generation to repository-scale assistance. However, existing benchmarks largely rely on bug reports from GitHub Issues, which often allow models to bypass genuine understanding via pattern matching on error logs. This misalignment under-measures Edit Bias, which refers to premature generation, where models prematurely propose code modifications instead of understanding the existing repository architecture. Furthermore, current LLM-as-a-Judge scalar scoring suffers from high variance and low interpretability. This work introduces RepoProbe, a novel benchmark for evaluating repository-level code understanding through open-ended Q&A using GitHub Discussions, which focuses on open-ended architectural inquiries rather than defect reporting. To ensure rigorous evaluation, we propose a Checklist-Based Verification Protocol that decomposes answers into atomic, verifiable facts, thereby replacing subjective ratings with objective verification. Our evaluation of state-of-the-art (SOTA) LLMs reveals a persistent gap between high clarity and evidence-grounded technical correctness. It also quantitatively confirms the prevalence of edit bias, in which models prioritize code generation instead of architectural analysis. Finally, we demonstrate that our verification protocol significantly improves evaluation reliability compared to traditional evaluations with scalar scoring.
@inproceedings{paperbot12599,
title = {RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists},
author = {Yuexi Yang and Alyssa Wu and Ji Luo and Richeng Xuan and Zhichao Hu and Yuhong Liu and Zhen Qin},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3837452}
}