一句话：揭示长尾代码分布使代码大模型在低频标签样本上性能下降30.0%至254.0%。
现有基于学习的技术，尤其是面向代码的大型语言模型，已广泛用于各类软件工程任务，但多数工作聚焦模型设计，较少关注数据集属性。本文对软件工程数据分布进行探索性研究，发现其通常呈长尾分布：少数类别拥有大量样本，而大量类别样本极少。作者在三个不同的软件工程任务上分析长尾分布对代码大模型性能的影响。实验结果表明，长尾分布显著影响模型效果：对于与低频标签相关的数据样本，代码大模型的性能比高频标签样本差30.0%到254.0%。该研究有助于理解长尾分布对流行代码大模型的影响，并为未来软件工程自动化的发展提供启示。
Learning-based techniques, especially advanced Large Language Models (LLMs) for code, have gained considerable popularity in various software engineering (SE) tasks. However, most existing works focus on designing better learning-based models and pay less attention to the properties of datasets. Learning-based models, including popular LLMs for code, heavily rely on data, and the data's properties (e.g., data distribution) could significantly affect their behavior. We conducted an exploratory study on the distribution of SE data and found that such data usually follows a skewed distribution (i.e., long-tailed distribution) where a small number of classes have an extensive collection of samples, while a large number of classes have very few samples. We investigate three distinct SE tasks and analyze the impacts of long-tailed distribution on the performance of LLMs for code. Our experimental results reveal that the long-tailed distribution has a substantial impact on the effectiveness of LLMs for code. Specifically, LLMs for code perform between 30.0% and 254.0% worse on data samples associated with infrequent labels compared to data samples of frequent labels. Our study provides a better understanding of the effects of long-tailed distributions on popular LLMs for code and insights for the future development of SE automation.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2309-03567,
title = {The Devil is in the Tails: How Long-Tailed Code Distributions Impact Large Language Models},
author = {Xin Zhou and Kisub Kim and Bowen Xu and Jiakun Liu and DongGyun Han and David Lo},
booktitle = {ASE},
year = {2023},
doi = {10.1109/ase56229.2023.00157}
}