一句话：通过 IDE 插件和一年真实使用数据，定量与定性评估三个公开代码补全语言模型，并对比在线与离线表现。
该研究关注代码补全语言模型缺乏真实场景评估的问题。作者开发开源 IDE 扩展 Code4Me，在线收集 1200 多名用户、超过一年、60 万余条有效补全数据，并用六个标准指标在十二种编程语言上评估三个公开模型；随后对 1690 个真实补全请求做定性分析，探究模型表现不佳的原因，还使用合成基准数据集和两种遮蔽策略比较在线与离线性能。结果揭示真实使用与离线基准之间的差异，为代码补全模型的实践评估提供数据与方法。
Transformer-based language models for automatic code completion have shown great promise so far, yet the evaluation of these models rarely uses real data. This study provides both quantitative and qualitative assessments of three public code language models when completing real-world code. We first developed an open-source IDE extension, Code4Me, for the online evaluation of the models. We collected real auto-completion usage data for over a year from more than 1200 users, resulting in over 600K valid completions. These models were then evaluated using six standard metrics across twelve programming languages. Next, we conducted a qualitative study of 1690 real-world completion requests to identify the reasons behind the poor model performance. A comparative analysis of the models' performance in online and offline settings was also performed, using benchmark synthetic datasets and two masking strategies.
@inproceedings{paperbot11273,
title = {Language Models for Code Completion: A Practical Evaluation},
author = {Maliheh Izadi and Jonathan Katzy and Tim Van Dam and Marc Otten and Razvan Mihai Popescu and Arie Van Deursen},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3639138}
}