一句话：提出 ContextCRBench——一个富上下文、细粒度的代码评审基准，含 6.79 万条数据，揭示现有 LLM 的明显局限。
现有 LLM 代码评审基准存在三重局限：缺少 issue 描述等语义上下文、验证不足导致数据质量偏低、粒度粗（文件或提交级）而忽略行级细节。本文提出 ContextCRBench：从热门仓库爬取超 15.37 万条 issue 与 PR；通过严格关联 issue-PR 对获得文本上下文，并抽取完整所在函数或类作为代码上下文；再经多阶段过滤（去除过期、格式不规范及由 LLM 分类器判定的低价值样本），最终得到 67,910 条富上下文条目。基准支持三种评测场景：hunk 级质量评估、行级缺陷定位与评审意见生成。对八个主流 LLM 的评测显示现有模型能力明显不足，且文本上下文比单纯代码上下文带来更大提升；在字节跳动作为自演化工具的奖励信号部署时，带来 61.98% 的性能提升。
While large language models (LLMs) have shown efficacy in automated code review, existing benchmarks suffer from three limitations: (1) a lack of rich semantic context like issue descriptions; (2) data quality issues due to insufficient validation; and (3) coarse granularity (file or commit level) that misses fine-grained, line-level nuances. To address these, we present ContextCRBench, a high-quality, context-rich benchmark designed for fine-grained evaluation of LLMs in code review tasks. Our construction pipeline consists of three main modules. First, the Raw Data Crawling module collects over 153.7k issues and PRs from the selected most popular repositories. Next, the Comprehensive Context Extraction module establishes rich context by rigorously linking issue-PR pairs for textual context and extracting the full surrounding function or class for code context. Finally, our Multi-stage Data Filtering module applies a series of checks to remove entries that are outdated, improperly formatted, or identified as low-value by an LLM-based classifier. This rigorous process yields the final benchmark of 67,910 entries. Each entry in our benchmark is enriched with both textual context and code context. Our benchmark supports three evaluation scenarios: hunk-level quality assessment, line-level defect localization, and review comment generation. Evaluations on eight popular LLMs reveal great limitations in current models, and textual context improves performance more than code context alone. Furthermore, when deployed at ByteDance as a reward signal for a self-evolving tool, ContextCRBench guided a 61.98% performance improvement, demonstrating its practical industrial utility. All scripts and datasets are available at https://github.com/kinesiatricssxilm14/ContextCRBench.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2511-07017,
title = {Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice},
author = {Ruida Hu and Xinchen Wang and Xin-Cheng Wen and Zhao Zhang and Bo Jiang and Pengfei Gao and Chao Peng and Cuiyun Gao},
booktitle = {FSE},
year = {2026},
doi = {10.1145/3803437.3805235}
}