一句话：通过专家构建的 HQCM 数据集微调小语言模型，在代码变更理解任务上可与大模型媲美甚至更优。
近期研究表明，直接提示语言模型在代码变更理解上比传统技术更有效，但现有方法依赖 GPT-4、Llama-13b 等昂贵或商业大模型，难以大规模部署。本文探索小规模语言模型（SLM）在保持与 LLM 相当或更优性能下的可行性。作者构建了由五位人类专家严格审查、修订和验证的高质量小数据集 HQCM，并用其微调先进的 7B 和 220M 参数 SLM，与参数量不少于 70B 的 LLM 及传统技术进行比较。评估确认了 HQCM 的益处，并表明微调后的 SLM 在变更摘要、变更分类和代码精化三个任务上均取得更优表现，适合安全、算力和成本受限的工业与边缘设备场景。
Recent studies indicate that traditional techniques for understanding code changes are not as effective as techniques that directly prompt language models (LMs). However, current LM-based techniques heavily rely on expensive, large LMs (LLMs) such as GPT-4 and Llama-13b, which are either commercial or prohibitively costly to deploy on a wide scale, thereby restricting their practical applicability. This paper explores the feasibility of deploying small LMs (SLMs) while maintaining comparable or superior performance to LLMs in code change understanding. To achieve this, we created a small yet high-quality dataset called HQCM which was meticulously reviewed, revised, and validated by five human experts. We fine-tuned state-of-the-art 7b and 220m SLMs using HQCM and compared them with traditional techniques and LLMs with ≥70b parameters. Our evaluation confirmed HQCM's benefits and demonstrated that SLMs, after finetuning by HQCM, can achieve superior performance in three change understanding tasks: change summarization, change classification, and code refinement. This study supports the use of SLMs in environments with security, computational, and financial constraints, such as in industry scenarios and on edge devices, distinguishing our work from the others.
@inproceedings{paperbot8308,
title = {Understanding Code Changes Practically with Small-Scale Language Models},
author = {Cong Li and Zhaogui Xu and Peng Di and Dongxia Wang and Zheng Li and Qian Zheng},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3694999}
}