一句话：系统评估现有AI代码检测工具并发现其泛化性差，提出基于微调LLM与AST特征的检测方法，F1达82.55。
随着大语言模型被广泛用于生成源代码，其代码质量问题与版权/许可侵权风险使检测代码是否由AI编写变得必要。本文首先对现有AI检测工具进行实证分析，发现它们在检测AI生成代码时表现不佳且缺乏足够泛化性，难以实际部署。为提升检测性能，作者提出多种方法，包括微调LLM，以及利用静态代码度量或从抽象语法树（AST）生成的代码嵌入进行机器学习分类。最佳模型超越了当前最先进的AI生成代码检测器GPTSniffer，取得82.55的F1分数。作者还对最佳模型进行消融研究，考察不同源代码特征对其性能的影响。
Artificial Intelligence (AI) techniques, especially Large Language Models (LLMs), have started gaining popularity among researchers and software developers for generating source code. However, LLMs have been shown to generate code with quality issues and also incurred copyright/licensing infringements. Therefore, detecting whether a piece of source code is written by humans or AI has become necessary. This study first presents an empirical analysis to investigate the effectiveness of the existing AI detection tools in detecting AI-generated code. The results show that they all perform poorly and lack sufficient generalizability to be practically deployed. Then, to improve the performance of AI-generated code detection, we propose a range of approaches, including fine-tuning the LLMs and machine learning-based classification with static code metrics or code embedding generated from Abstract Syntax Tree (AST). Our best model outperforms state-of-the-art AI-generated code detector (GPTSniffer) and achieves an F1 score of 82.55. We also conduct an ablation study on our best-performing model to investigate the impact of different source code features on its performance.
DOI 原文 ·
@inproceedings{paperbot8742,
title = {An Empirical Study on Automatically Detecting AI-Generated Source Code: How Far are We?},
author = {Hyunjae Suh and Mahan Tafreshipour and Jiawei Li and Adithya Bhattiprolu and Iftekhar Ahmed},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00064}
}