一句话：提出基于扰动的记忆优势度量，比较8个开源代码LLM在19个基准上对训练见过与新颖代码的性能差距。
代码LLM训练数据不透明，导致数据泄露与记忆化难以检测和评估。本文采用基于扰动的方法，通过比较模型在可能于训练中见过的数据与新颖输入上的性能差距，量化其“记忆化优势”。作者系统分析了8个开源代码LLM在19个基准数据集上的表现，覆盖代码生成、代码理解、安全漏洞检测和缺陷识别四类任务。结果显示，不同模型的敏感性差异显著：StarCoder在APPS等基准上敏感性可达0.8，而QwenCoder多数低于0.4；代码摘要任务敏感性较低（<0.3），测试生成较高（0.4-0.7）。常被怀疑数据泄露的CVEFixes和Defects4J反而记忆优势很低，CVEFixes低于0.1，Defects4J为0.2-0.4，低于其他程序修复基准（0.5-0.8），表明模型可能更多是在泛化而非记忆。研究强调需要更稳健的评估框架，尤其安全相关领域。
The lack of transparency regarding the code datasets used during LLM training creates substantial challenges in detecting, evaluating, and mitigating data leakage. This paper applies a perturbation-based approach to quantify the “memorization advantage” of LLMs across various coding tasks by measuring the performance gap between a model’s handling of data it has likely encountered during training versus novel inputs. Our comprehensive analysis examines 8 open-source code LLMs across 19 benchmark datasets spanning four distinct categories: standard code generation, code understanding, security vulnerability detection, and bug identification. The results reveal significant variations in sensitivity patterns, with models like StarCoder exhibiting substantially higher sensitivity scores (up to 0.8) on certain benchmarks like APPS compared to models like QwenCoder, which maintained consistently lower values (< 0.4) across most benchmarks, suggesting fundamental differences in their generalization process and their learned knowledge. Different task categories also showed distinct patterns, with code summarization demonstrating low sensitivity (< 0.3) and test generation tasks exhibiting significantly higher values (0.4-0.7, p < 0.001). Interestingly, our analysis of the widely-used CVEFixes and Defects4J benchmarks, frequently suspected of data leakage in the research community, reveals unexpectedly low memorization advantage scores across all models. Defects4J demonstrated significantly lower sensitivity (0.2-0.4, p < 0.01) compared to other program repair benchmarks (0.5-0.8), while CVEFixes showed consistently low values below 0.1. These findings challenge prevailing concerns about these datasets’ validity for evaluating code LLMs and suggest that models may be effectively generalizing from this data rather than merely memorizing it. Our findings provide critical insights into the generalization capabilities of code LLMs and emphasize the need for more robust evaluation frameworks, particularly in security-related domains where the widest range of sensitivity distributions (from < 0.1 to > 0.8) indicates variable generalization challenges.
@inproceedings{abs-2604-13997,
title = {Quantifying Memorization Advantage in Code LLMs},
author = {Alberick Euraste Djire and Abdoul Kader Kaboré and Jordan Samhi and Earl Barr and Jacques Klein and Tegawendé F. Bissyandé},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3764554}
}