一句话：系统考察代码生成模型训练数据的质量属性与处理策略，揭示数据设计选择对模型性能的显著影响。
尽管语言模型数量激增，其训练数据集的透明度仍然不足：安全顾虑常被引用为不公开的理由，但识别高质量训练数据对模型性能至关重要。已有工作多聚焦于改进模型本身，数据集质量这一环节仍少有探索。本文针对代码生成模型，系统研究数据质量属性与处理策略：从元数据、统计特征、数据质量问题、意图与代码之间的语义相关性以及数据集设计选择等多个维度展开分析，并通过操纵这些特征来考察其对模型性能的影响。研究发现，数据集的设计选择会显著影响代码生成模型的性能；意图与代码之间的语义相关性同样会带来影响，但程度因情形而异。该工作为代码 LLM 的数据集构建与优化提供了经验依据。
Despite the proliferation of language models, a lack of transparency persists regarding the training datasets used. Security concerns are often cited, but identifying high-quality training data is crucial for optimal model performance. Yet, while significant efforts have been made to improve model performance, dataset quality remains an under-explored area. Our study addresses this gap by comprehensively investigating data-quality properties and processing strategies used to train code generation models. We focus on identifying dataset features that impact model performance and leverage these insights to optimize datasets and enhance model efficacy. Our approach involves a multifaceted analysis encompassing metadata, statistics, data quality issues, semantic correlations between intent and code, and design choices. By manipulating these features, we explore their influence on model performance. Our findings reveal that dataset design choices significantly impact the performance of code generation models. Additionally, semantic correlations between intent and code can also affect performance, although to varying degrees.CCS CONCEPTS• Software and its engineering → Automatic programming; • Computing methodologies → Information extraction.
@inproceedings{KimKP0CWSTKB24,
title = {DataRecipe --- How to Cook the Data for CodeLLM?},
author = {Kisub Kim and Jounghoon Kim and Byeongjo Park and Dongsun Kim and Chun Yong Chong and Yuan Wang and Tiezhu Sun and Daniel Tang and Jacques Klein and Tegawende F. Bissyande},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695593}
}