一句话：对 code2vec 方法名推荐进行更真实设置下的实证评估，揭示其局限，并提出启发式方法显著超越。
高质量方法名对程序可读性与可维护性至关重要，但自动推荐仍具挑战。本文对当前最优的 code2vec 方法名推荐开展实证研究，在更真实数据集与设置下重新评估，发现更换数据集影响不大，但更真实设置会显著降低性能。对成功推荐的分析表明，约 48.3% 的接受推荐来自 getter/setter 方法，19.2% 的成功推荐名可直接从方法体复制；开发者人工标注命名难度后，code2vec 在真正需要命名时很少有效。作者进而提出基于启发式规则的方法名推荐，在大规模数据集上 precision 和 recall 分别提升 65.25% 和 22.45%，显著优于机器学习方法，表明该方向仍有很长的路要走。
High quality method names are critical for the readability and maintainability of programs. However, constructing concise and consistent method names is often challenging, especially for inexperienced developers. To this end, advanced machine learning techniques have been recently leveraged to recommend method names automatically for given method bodies/implementation. Recent large-scale evaluations also suggest that such approaches are accurate. However, little is known about where and why such approaches work or don't work. To figure out the state of the art as well as the rationale for the success/failure, in this paper we conduct an empirical study on the state-of-the-art approach code2vec. We assess code2vec on a new dataset with more realistic settings. Our evaluation results suggest that although switching to new dataset does not significantly influence the performance, more realistic settings do significantly reduce the performance of code2vec. Further analysis on the successfully recommended method names also reveals the following findings: 1) around half (48.3%) of the accepted recommendations are made on getter/setter methods; 2) a large portion (19.2%) of the successfully recommended method names could be copied from the given bodies. To further validate its usefulness, we ask developers to manually score the difficulty in naming methods they developed. Code2vec is then applied to such manually scored methods to evaluate how often it works in need. Our evaluation results suggest that code2vec rarely works when it is really needed. Finally, to intuitively reveal the state of the art and to investigate the possibility of designing simple and straightforward alternative approaches, we propose a heuristics based approach to recommending method names. Evaluation results on large-scale dataset suggest that this simple heuristics-based approach significantly outperforms the state-of-the-art machine learning based approach, improving precision and recall by 65.25% and 22.45%, respectively. The comparison suggests that machine learning based recommendation of method names may still have a long way to go.
DOI 原文 ·
@inproceedings{paperbot5270,
title = {Machine Learning Based Recommendation of Method Names: How Far are We},
author = {Lin Jiang and Hui Liu and He Jiang},
booktitle = {ASE},
year = {2019},
doi = {10.1109/ase.2019.00062}
}