一句话：首个系统性实证研究：大规模评测 LLM 生成 fuzz driver 的有效性、提示策略与工业界差距。
模糊测试驱动器（fuzz driver）是库 API 模糊测试的关键，但自动生成正确、鲁棒的 API 使用代码十分困难。基于 LLM 的文本化生成方法比传统程序分析生成器更通用、可读，但其有效性与挑战尚缺乏系统认识。本文开展首个深入研究：构建覆盖 30 个常用 C 项目、共 86 个问题的数据集，设计 6 种提示策略，在 5 个先进 LLM 与 5 种温度设置下评估 736,430 个 driver（8.5 亿 token），并与工业界 driver 进行 3.75 CPU 年的大规模模糊测试对比。结果发现：LLM 生成 driver 的模糊测试效果已接近工业水平，但对 API 细节复杂的场景仍力不从心；重复查询、示例查询与迭代查询三类提示设计有益；扩展 API 使用范围、引入语义 oracle 检测逻辑缺陷仍有很大提升空间。相关经验已用于改进 OSS-Fuzz-Gen。
Fuzz drivers are essential for library API fuzzing. However, automatically generating fuzz drivers is a complex task, as it demands the creation of high-quality, correct, and robust API usage code. An LLM-based (Large Language Model) approach for generating fuzz drivers is a promising area of research. Unlike traditional program analysis-based generators, this text-based approach is more generalized and capable of harnessing a variety of API usage information, resulting in code that is friendly for human readers. However, there is still a lack of understanding regarding the fundamental issues on this direction, such as its effectiveness and potential challenges. To bridge this gap, we conducted the first in-depth study targeting the important issues of using LLMs to generate effective fuzz drivers. Our study features a curated dataset with 86 fuzz driver generation questions from 30 widely-used C projects. Six prompting strategies are designed and tested across five state-of-the-art LLMs with five different temperature settings. In total, our study evaluated 736,430 generated fuzz drivers, with 0.85 billion token costs ($8,000+ charged tokens). Additionally, we compared the LLM-generated drivers against those utilized in industry, conducting extensive fuzzing experiments (3.75 CPU-year). Our study uncovered that: 1) While LLM-based fuzz driver generation is a promising direction, it still encounters several obstacles towards practical applications; 2) LLMs face difficulties in generating effective fuzz drivers for APIs with intricate specifics. Three featured design choices of prompt strategies can be beneficial: issuing repeat queries, querying with examples, and employing an iterative querying process; 3) While LLM-generated drivers can yield fuzzing outcomes that are on par with those used in the industry, there are substantial opportunities for enhancement, such as extending contained API usage, or integrating semantic oracles to facilitate logical bug detection. Our insights have been implemented to improve the OSS-Fuzz-Gen project, facilitating practical fuzz driver generation in industry.
@inproceedings{paperbot8569,
title = {How Effective Are They? Exploring Large Language Model Based Fuzz Driver Generation},
author = {Cen Zhang and Yaowen Zheng and Mingqiang Bai and Yeting Li and Wei Ma and Xiaofei Xie and Yuekang Li and Limin Sun and Yang Liu},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3680355}
}