一句话：用三级代码混淆构建基准 OBFusEval，揭示 LLM 在未见代码上测试通过率最多下降 62.5%。
现有 LLM 代码生成评测存在“熟悉即专业”的错觉：数据集中的目标代码可能在训练阶段已被大量暴露，且案例时效性与依赖可用性不足，导致评测高估模型真实能力。为尽可能用模型未接触过的代码进行评测，本文借鉴代码混淆思想，在保证功能与输出不变的前提下从不同层面改写代码。作者从五个真实项目中收集 1354 个原始用例（含函数描述与代码），采用符号、结构、语义三级策略对描述、代码和上下文依赖进行混淆，构建基准 OBFusEval。随后在四个 LLM 上评测，并用项目官方测试套件检验生成代码，同时比较不同混淆策略的效果。结果显示，混淆后测试通过率平均下降比例最高达 62.5%，说明模型在熟悉代码上的表现难以代表其真实代码生成能力。
Recently, large language models (LLMs) have shown strong potential in code generation tasks. However, there are still gaps before they can be fully applied in actual software development processes. Accurately assessing the code generation capabilities of large language models has become an important basis for evaluating and improving the models. Some existing works have constructed datasets to evaluate the capabilities of these models. However, the current evaluation process may encounter the illusion of “Specialist in Familiarity”, primarily due to three gaps: the exposure of target code, case timeliness, and dependency availability. The fundamental reason for these gaps is that the code in current datasets may have been extensively exposed and exercised during the training phase, and due to the continuous training and development of LLM, their timeliness has been severely compromised. The key to solve the problem is to, as much as possible, evaluate the LLMs using code that they have not encountered before. Thus, the fundamental idea in this paper is to draw on the concept of code obfuscation, changing code at different levels while ensuring the functionality and output. To this end, we build a code-obfuscation based benchmark OBFusEvAL. We first collect 1,354 raw cases from five real-world projects, including function description and code. Then we use three-level strategy (symbol, structure and semantic) to obfuscate descriptions, code and context dependencies. We evaluate four LLMs on Obfu-sevaland compared the effectiveness of different obfuscation strategy. We use official test suites of these projects to evaluate the generated code. The results show that after obfuscation, the average decrease ratio of test pass rate can up to 62.5%.
DOI 原文 ·
@inproceedings{paperbot8829,
title = {Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar},
author = {Yuanliang Zhang and Yifan Xie and Shanshan Lit and Ke Liu and Chong Wang and Zhouyang Jia and Xiangbing Huang and Jie Song and Chaopeng Luo and Zhizheng Zheng and Rulin Xu and Yitong Liu and Si Zheng and Xiangke Liao},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00082}
}