一句话：首次系统评测 LLM 检测 Token 不一致性缺陷的能力，并提出级联检测系统 LineBreaker，提升精度、召回与可扩展性。
Token 不一致性缺陷（TIB）指语法合法但语义错误的代码 token 误用，如变量误用和错误函数调用。这类缺陷隐蔽且依赖上下文，传统静态分析与动态测试难以发现。本文首次系统测量 LLM（如 GPT-4）检测 TIB 的能力，发现其精度受无关片段干扰、成本高且难以扩展。为此提出 LineBreaker：级联式 TIB 检测系统，先用小型、代码专用且高效的语言模型过滤大量不太可能含 TIB 的片段，以提升精度、召回与可扩展性。在 154 个 Python 与 C GitHub 仓库（均超 1,000 stars）上评估，发现 123 个新缺陷，其中 45% 可被利用破坏程序功能；提交的 69 个修复中 41 个已确认或合并。
Token-inconsistency bugs (TIBs) involve the misuse of syntactically valid yet incorrect code tokens, such as misused variables and erroneous function invocations, which can often lead to software bugs. Unlike simple syntactic bugs, TIBs occur at the semantic level and are subtle - sometimes they remain undetected for years. Traditional detection methods, such as static analysis and dynamic testing, often struggle with TIBs due to their versatile and context-dependent nature. However, advancements in large language models (LLMs) like GPT-4 present new opportunities for automating TIB detection by leveraging these models’ semantic understanding capabilities.This paper reports the first systematic measurement of LLMs’ capabilities in detecting TIBs, revealing that while GPT-4 shows promise, it exhibits limitations in precision and scalability. Specifically, its detection capability is undermined by the model’s tendency to focus on the code snippets that do not contain TIBs; its scalability concern arises from GPT-4’s high cost and the massive amount of code requiring inspection. To address these challenges, we introduce LineBreaker, a novel and cascaded TIB detection system. LineBreaker leverages smaller, codespecific, and highly efficient language models to filter out large numbers of code snippets unlikely to contain TIBs, thereby significantly enhancing the system’s performance in terms of precision, recall, and scalability. We evaluated LineBreaker on 154 Python and C GitHub repositories, each with over 1,000 stars, uncovering 123 new flaws, 45% of which could be exploited to disrupt program functionalities. Out of our 69 submitted fixes, 41 have already been confirmed or merged.
DOI 原文 ·
@inproceedings{paperbot9555,
title = {LineBreaker: Finding Token-Inconsistency Bugs with Large Language Models},
author = {Hongbo Chen and Yifan Zhang and Xing Han and Tianhao Mao and Huanyao Rong and Yuheng Zhang and XiaoFeng Wang and Luyi Xing and Xun Chen and Hang Zhang},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00079}
}