一句话：首次系统评测模型编辑修复 LLMs4Code 代码知识，提出 CLMEEval 基准与增强方法 A-GRACE。
大型代码语言模型（LLMs4Code）在编码任务中表现优异，但难免包含错误或过时的代码知识，而重新训练成本高昂。模型编辑为高效修正 LLM 知识提供了新途径，但尚缺乏面向 LLMs4Code 的系统比较研究。本文首次系统研究将先进模型编辑技术用于修复 LLMs4Code 的不准确知识，并提出基准 CLMEEval，包含 CoNaLa-Edit（21K+ 代码生成样本）和 CodeSearchNet-Edit（16K+ 代码摘要样本）。基于该基准，作者在 CodeLlama (7B)、CodeQwen1.5 (7B) 和 Stable-Code (3B) 上评估六种模型编辑技术。结果表明，基于外部记忆的 GRACE 在编辑有效性和特异性上最佳，但泛化到语义相同输入是普遍挑战。作者进一步提出 A-GRACE，引入对比学习更好地捕捉输入语义，在保持有效性和特异性的同时显著提升泛化能力。
Large Language Models for Code (LLMs4Code) have been found to exhibit outstanding performance in the software engineering domain, especially the remarkable performance in coding tasks. However, even the most advanced LLMs4Code can inevitably contain incorrect or outdated code knowledge. Due to the high cost of training LLMs4Code, it is impractical to re-train the models for fixing these problematic code knowledge. Model editing is a new technical field for effectively and efficiently correcting erroneous knowledge in LLMs, where various model editing techniques and benchmarks have been proposed recently. Despite that, a comprehensive study that thoroughly compares and analyzes the performance of the state-of-the-art model editing techniques for adapting the knowledge within LLMs4Code across various code-related tasks is notably absent. To bridge this gap, we perform the first systematic study on applying state-of-the-art model editing approaches to repair the inaccuracy of LLMs4Code. To that end, we introduce a benchmark named CLMEEval, which consists of two datasets, i.e., CoNaLa-Edit (CNLE) with 21K+ code generation samples and CodeSearchNet-Edit (CSNE) with 16K+ code summarization samples. With the help of CLMEEval, we evaluate six advanced model editing techniques on three LLMs4Code: CodeLlama (7B), CodeQwen1.5 (7B), and Stable-Code (3B). Our findings include that the external memorization-based GRACE approach achieves the best knowledge editing effectiveness and specificity (the editing does not influence untargeted knowledge), while generalization (whether the editing can generalize to other semantically-identical inputs) is a universal challenge for existing techniques. Furthermore, building on in-depth case analysis, we introduce an enhanced version of GRACE called A-GRACE, which incorporates contrastive learning to better capture the semantics of the inputs. Results demonstrate that A-GRACE notably enhances generalization while maintaining similar levels of effectiveness and specificity compared to the vanilla GRACE.
DOI 原文 ·
@inproceedings{paperbot8766,
title = {Model Editing for LLMs4Code: How Far are we?},
author = {Xiaopeng Li and Shangwen Wang and Shasha Li and Jun Ma and Jie Yu and Xiaodong Liu and Jing Wang and Bin Ji and Weimin Zhang},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00049}
}