一句话：系统研究发现现有安全代码生成技术常以牺牲功能正确性换取安全性，且单一静态分析器 CodeQL 的评估存在盲区。
大语言模型生成的代码常含漏洞，已有多种安全代码生成方法，但其评估方案存在隐患：多数研究用不同数据集分别评估安全性与功能正确性，且仅依赖单一静态分析器 CodeQL 检测漏洞。本文对四种最先进的安全代码生成技术展开系统研究，将安全检测与功能验证施加于同一份生成代码，并联合评估两方面；同时采用三种主流静态分析器与两个 LLM 共同识别潜在漏洞。研究发现，现有技术往往通过牺牲生成代码的功能性来提升安全性，当安全与功能联合评估时整体表现有限，许多技术甚至使基础 LLM 的性能下降超过 50%。进一步检查表明，这些技术常常直接删除含漏洞的代码行，或生成与任务无关的“垃圾代码”。此外，常用的 CodeQL 漏检若干漏洞，进一步掩盖了实际安全改进。本文为未来更严格、更全面的安全代码生成评估提供了指南。
Large language models (LLMs) are widely used in software development. However, the code generated by LLMs often contains vulnerabilities. Several secure code generation methods have been proposed to address this issue, but their current evaluation schemes leave several concerns unaddressed. Specifically, most existing studies evaluate security and functional correctness separately, using different datasets. That is, they assess vulnerabilities using security-related code datasets while validating functionality with general code datasets. In addition, prior research primarily relies on a single static analyzer, CodeQL, to detect vulnerabilities in generated code, which limits the scope of security evaluation. In this work, we conduct a comprehensive study to systematically assess the improvements introduced by four state-of-the-art secure code generation techniques. Specifically, we apply both security inspection and functionality validation to the same generated code and evaluate these two aspects together. We also employ three popular static analyzers and two LLMs to identify potential vulnerabilities in the generated code. Our study reveals that existing techniques often compromise the functionality of generated code to enhance security. Their overall performance remains limited when evaluating security and functionality together. In fact, many techniques even degrade the performance of the base LLM by more than 50%. Our further inspection reveals that these techniques often either remove vulnerable lines of code entirely or generate “garbage code” that is unrelated to the intended task. Moreover, the commonly used static analyzer CodeQL fails to detect several vulnerabilities, further obscuring the actual security improvements achieved by existing techniques. Our study serves as a guideline for a more rigorous and comprehensive evaluation of secure code generation performance in future work.