一句话：提出 RepoSim 基准，通过回放 IDE 用户行为、模拟仓库演化来评估代码补全提示策略与用户接受率。
现有代码补全评估多依赖静态代码基准，忽略了真实的人机交互与仓库演化过程。本文提出 RepoSim，一个面向代码补全任务的基准框架：它采集 IDE 插件中的用户行为数据，通过记录与回放用户操作，为评估提供贴近真实的编程上下文，从而支持评估更复杂的提示策略，例如利用最近访问的文件与用户编辑历史。RepoSim 还提出一种基于用户对预测接受或拒绝的新度量，提供以用户为中心的评估标准。初步评估表明，将用户近期编辑历史加入提示可显著提升 LLM 生成代码的质量，说明时间上下文在代码补全中的重要性。总体而言，RepoSim 为代码补全性能评估提供了一个更真实、更聚焦用户的基准框架。
Large language models (LLMs) have revolutionized code completion tasks. IDE plugins such as MarsCode can generate code recommendations, saving developers significant time and effort. However, current evaluation methods for code completion are limited by their reliance on static code benchmarks, which do not consider human interactions and evolving repositories. This paper proposes RepoSim, a novel benchmark designed to evaluate code completion tasks by simulating the evolving process of repositories and incorporating user behaviors. RepoSim leverages data from an IDE plugin, by recording and replaying user behaviors to provide a realistic programming context for evaluation. This allows for the assessment of more complex prompt strategies, such as utilizing recently visited files and incorporating user editing history. Additionally, RepoSim proposes a new metric based on users’ acceptance or rejection of predictions, offering a user-centric evaluation criterion. Our preliminary evaluation demonstrates that incorporating users’ recent edit history into prompts significantly improves the quality of LLM-generated code, highlighting the importance of temporal context in code completion. RepoSim represents a significant advancement in benchmarking tools, offering a realistic and user-focused framework for evaluating code completion performance.
@inproceedings{0002WLLJXWLY24,
title = {RepoSim: Evaluating Prompt Strategies for Code Completion via User Behavior Simulation},
author = {Chao Peng and Qinyun Wu and Jiangchao Liu and Jierui Liu and Bo Jiang and Mengqian Xu and Yinghao Wang and Xia Liu and Ping Yang},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695299}
}