一句话：提出基于真实仓库、用变异测试增强测试套件的代码补全评测基准 RepoMasterEval。
现有代码补全评测基准多聚焦函数或类级别，并依赖文本描述提示，但真实开发中常无此类描述，补全可发生在函数或代码块中间，导致基准与实用场景错位。本文提出 RepoMasterEval：从带测试套件的真实仓库中遮盖代码片段（ground truth）构造评测数据，并用变异测试度量测试用例有效性，对低变异分数套件手工补充用例，以提升对模型生成代码的测试准确性。对 10 个先进模型的实证评估表明，测试增强对基准准确性至关重要，该基准能反映真实场景下的模型性能差异；部署还显示其可为训练提供准确反馈，且得分与实际表现高度相关。
With the growing reliance on automated code completion tools in software development, the need for comprehensive evaluation benchmarks has become critical. Existing benchmarks focus more on code completion in function and class level by pro-viding text descriptions to prompt the model. By contrast, such descriptive prompt is commonly unavailable in real development and code completion can occur in wider range of situations such as in the middle of a function or a code block. These limitations makes existing evaluation benchmarks poorly align with the practical scenarios of code completion tools. In this paper, we propose RepoMasterEval, a novel benchmark for evaluating code completion models constructed from real-world repositories. Each benchmark datum is generated by masking a code snippet (ground truth) from one source code file with existing test suites. To improve test accuracy of model generated code, we employ mutation testing to measure the effectiveness of the test cases and we manually crafted new test cases for those test suites with low mutation score. Our empirical evaluation on 10 state-of-the-art models shows that test argumentation is critical in improving the accuracy of the benchmark and RepoMasterEval is able to report variance in model performance in real-world scenarios. The deployment of RepoMasterEval also revealed that the benchmark is useful to give accurate feedback during model training and the score is in high correlation with the model’s performance in practice.
DOI 原文 ·
@inproceedings{paperbot9559,
title = {RepoMasterEval: Evaluating Code Completion via Real-World Repositories},
author = {Qinyun Wu and Chao Peng and Pengfei Gao and Ruida Hu and Haoyu Gan and Bo Jiang and Jinhe Tang and Zhiwen Deng and Zhanming Guan and Cuiyun Gao and Xia Liu and Ping Yang},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00304}
}