一句话：首次直接用大语言模型生成与变异深度学习库输入程序进行模糊测试，无需训练即超越现有工具。
深度学习库（如 TensorFlow、PyTorch）为下游系统提供 API，其缺陷检测至关重要。但传统模糊测试在此领域效果有限：生成的输入程序不仅需满足 Python 等宿主语言的语法与语义，还必须满足 DL API 的输入与张量形状约束。为此，作者提出 TitanFuzz，首个直接利用大语言模型（LLM）生成模糊测试输入程序的方法。其核心洞察是：现代 LLM 在训练语料中已包含大量调用 DL API 的代码片段，因而隐式学习到了语言语法语义与复杂的 DL API 约束。方法上，TitanFuzz 同时使用生成式与填充式 LLM（如 Codex、InCoder）来生成和变异合法且多样的输入 DL 程序。实验表明，在 TensorFlow/PyTorch 上其代码覆盖率比当时最先进的模糊测试器分别高出 30.38% 与 50.84%，并检出 65 个缺陷，其中 44 个已被确认为此前未知缺陷。
Deep Learning (DL) systems have received exponential growth in popularity and have become ubiquitous in our everyday life. Such systems are built on top of popular DL libraries, e.g., TensorFlow and PyTorch which provide APIs as building blocks for DL systems. Detecting bugs in these DL libraries is critical for almost all downstream DL systems in ensuring effectiveness/safety for end users. Meanwhile, traditional fuzzing techniques can be hardly effective for such a challenging domain since the input DL programs need to satisfy both the input language (e.g., Python) syntax/semantics and the DL API input/shape constraints for tensor computations. To address these limitations, we propose TitanFuzz – the first approach to directly leveraging Large Language Models (LLMs) to generate input programs for fuzzing DL libraries. LLMs are titanic models trained on billions of code snippets and can autoregressively generate human-like code snippets. Our key insight is that modern LLMs can also include numerous code snippets invoking DL library APIs in their training corpora, and thus can implicitly learn both language syntax/semantics and intricate DL API constraints for valid DL program generation. More specifically, we use both generative and infilling LLMs (e.g., Codex/InCoder) to generate and mutate valid/diverse input DL programs for fuzzing. Our experimental results demonstrate that TitanFuzz can achieve 30.38%/50.84% higher code coverage than state-of-the-art fuzzers on TensorFlow/PyTorch. Furthermore, TitanFuzz is able to detect 65 bugs, with 44 already confirmed as previously unknown bugs. This paper demonstrates that modern titanic LLMs can be leveraged to directly perform both generation-based and mutation-based fuzzing studied for decades, while being fully automated, generalizable, and applicable to domains challenging for traditional approaches (such as DL systems). We hope TitanFuzz can stimulate more work in this promising direction of LLMs for fuzzing.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{DengXPY023,
title = {Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models},
author = {Yinlin Deng and Chunqiu Steven Xia and Haoran Peng and Chenyuan Yang and Lingming Zhang},
booktitle = {ISSTA},
year = {2023},
doi = {10.1145/3597926.3598067}
}