一句话：提出 RACE-Bench，用可执行补丁验证与结构化中间推理轨迹双重评估仓库级代码智能体的特性添加能力。
现有仓库级代码智能体评测多以黑盒方式进行，仅看最终补丁能否通过测试，难以揭示其推理过程与失败根源。本文提出 RACE-Bench：一个面向仓库级特性添加任务的推理增强基准，包含来自 12 个开源仓库的 528 个真实特性添加实例，每个实例同时配备可执行的补丁验证与结构化中间参考推理，覆盖问题理解、文件定位、实现任务与步骤分解四个层面。基于该设计，论文提出双轨评估框架，联合衡量补丁正确性与中间推理同开发者认可参考轨迹的对齐程度。作者在三个代表性仓库级代码智能体上评测，Resolved Rate 介于 29% 至 70%；推理层面分析显示智能体能较好理解高层意图，但在把意图转化为具体实现步骤时性能显著下降，且可应用却测试失败的补丁具有更低的参考推理要素召回率（低 35.7%）与更多无依据推理要素（过度预测高 94.1%）。
Repository-level code agents have shown strong promise in real-world feature addition tasks, making reliable evaluation of their capabilities increasingly important. However, existing benchmarks primarily evaluate these agents as black boxes based on final test correctness, providing limited insight into how they reason and where failures arise. To address this limitation, we introduce RACE-Bench, a reasoning-augmented benchmark for evaluating code agents on repository-level feature addition tasks. RACE-Bench contains 528 real-world feature addition instances from 12 open-source repositories. Each instance is paired with executable patch verification and structured intermediate reference reasoning covering issue understanding, file localization, implementation tasks, and step decomposition. Based on this design, we introduce a dual-track evaluation framework that jointly measures patch correctness and intermediate reasoning alignment with developer-accepted reference trajectories. We evaluate three representative repository-level code agents on RACE-Bench. On the full benchmark, Resolved Rate ranges from 29% to 70% across different agents. Our reasoning-level analysis further shows that while current agents perform well at understanding high-level intent, their performance degrades substantially when translating intent into concrete implementation steps. We also find patches that can be applied but still fail the tests cover fewer reference-reasoning elements (35.7% lower recall) and contain more unsupported reasoning elements (94.1% higher over-prediction) than successful patches. These findings highlight the importance of evaluating repository-level code agents beyond final patch correctness by examining the quality of their reasoning processes.
@inproceedings{paperbot12421,
title = {RACE-Bench: A Reasoning-Augmented Benchmark for Repository-Level Code Agents on Feature Addition},
author = {Shuhan Liu and Zhiyi Zhao and Xing Hu and Kui Liu and Xiaohu Yang and Xin Xia},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3837426}
}