一句话：构建API推荐基准，系统评测九种大型代码模型在“何时用、用哪个、如何用”三个场景下的能力。
现有API推荐研究多聚焦于“用哪个API”，忽视了开发者对“何时使用”与“如何使用”API的辅助需求。本文构建了覆盖853个流行Java项目、176个API的基准，并划分三个评测场景：确定使用位置与时机的“何时用”、从给定库中选取合适API的“用哪个”、为给定API预测参数的“如何用”。作者选取九种不同规模的流行大型代码模型（LCM）进行评测，并分析上下文选择的影响。结果显示，LCM在“如何用”表现最佳、“何时用”最差，两者平均差距达34%，表明“何时用”更具挑战；丰富上下文能显著提升性能，使小模型在提供上下文时超越未提供上下文的大二十倍模型。最后给出对研究者与开发者的启示，为API推荐任务奠定基础。
API suggestion is a critical task in modern software development, assisting programmers by predicting and recommending third-party APIs based on the current context. Recent advancements in large code models (LCMs) have shown promise in the API suggestion task. However, they mainly focus on suggesting which APIs to use, ignoring that programmers may demand more assistance while using APIs in practice including when to use the suggested APIs and how to use the APIs. To mitigate the gap, we conduct a systematic evaluation of LCMs for the API suggestion task in the paper.To facilitate our investigation, we first build a benchmark that contains a diverse collection of code snippets, covering 176 APIs used in 853 popular Java projects. Three distinct scenarios in the API suggestion task are then considered for evaluation, including (1) "when to use", which aims at determining the desired position and timing for API usage; (2) "which to use", which aims at identifying the appropriate API from a given library; and (3) "how to use", which aims at predicting the arguments for a given API. The consideration of the three scenarios allows for a comprehensive assessment of LCMs’ capabilities in suggesting APIs for developers. During the evaluation, we choose nine popular LCMs with varying model sizes for the three scenarios. We also perform an in-depth analysis of the influence of context selection on the model performance. Our experimental results reveal multiple key findings. For instance, LCMs present the best performance in the "how to use" scenario while performing the worst in the "when to use" scenario, e.g., the average performance gap of LCMs between "when to use" and "how to use" scenarios achieves 34%, indicating that the "when to use" scenario is more challenging. Furthermore, enriching context information substantially improves the model performance. Specifically, by incorporating the contexts, smaller-sized LCMs can outperform those twenty times larger models without the contexts provided. Based on these findings, we finally provide insights and implications for researchers and developers, which can lay the groundwork for future advancements in the API suggestion task.CCS CONCEPTS• Software and its engineering → Software development techniques;
@inproceedings{abs-2409-13178,
title = {A Systematic Evaluation of Large Code Models in API Suggestion: When, Which, and How},
author = {Chaozheng Wang and Shuzheng Gao and Cuiyun Gao and Wenxuan Wang and Chun Yong Chong and Shan Gao and Michael R. Lyu},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695004}
}