一句话：基于超过12.5 CPU年实验评估AFL++变异算子，发现禁用某些变异算子可提升覆盖多样性与找bug速度。
灰盒模糊测试器通过变异输入并依据覆盖率保留有趣输入来演化语料库，但其变异设计决策对性能的影响往往难以预测，部分配置还硬编码在实现中。本文对流行的 AFL++ 模糊测试器所使用的变异算子集合开展超过 12.5 CPU 年的大规模实证评估，涵盖 havoc 阶段、splicing 以及若干未暴露配置。作者提出一种使用严谨统计方法衡量不同模糊测试器在分支覆盖与缺陷检测方面行为多样性的方法论。关键发现是：在多个目标程序上，禁用某些变异算子会产生覆盖不同代码行、暴露不同缺陷的输入；令人意外的是，禁用某些变异器还能带来更多样的覆盖，并使模糊测试器更快发现更多缺陷。作者呼吁更深入地研究模糊测试器中看似简单的设计决策，并鼓励开发者向用户暴露更多相关配置参数。
Over the past decade, hundreds of fuzzers have been published in top-tier security and software engineering conferences. Fuzzers are used to automatically test programs, ideally creating high-coverage input corpora and finding bugs. Modern “greybox” fuzzers evolve a corpus of inputs by applying mutations to inputs and then executing those new inputs while collecting coverage. New inputs that are “interesting” (e.g. reveal new coverage) are saved to the corpus. Given their non-deterministic nature, the impact of each design decision on the fuzzer’s performance can be difficult to predict. Some design decisions (e.g., ” Should the fuzzer perform deterministic mutations of inputs? ”) are exposed to end-users as configuration flags, but others (e.g., ” What kinds of random mutations to apply to inputs?”) are typically baked into the fuzzer code itself. This paper describes our over 12.5-CPU-year evaluation of the set of mutation operators employed by the popular AFL++ fuzzer, including the havoc phase, splicing, and, exploring the impact of adjusting some of those unexposed configurations. In this experience paper, we propose a methodology for determining different fuzzers’ behavioral diversity with respect to branch coverage and bug detection using rigorous statistical methods. Our key finding is that, across a range of targets, disabling certain mutation operators (some of which were previously “baked-in” to the fuzzer) resulted in inputs that cover different lines of code and reveal different bugs. A surprising result is disabling certain mutators leads to more diverse coverage and allows the fuzzer to find more bugs faster. We call for researchers to investigate seemingly simple design decisions in fuzzers more thoroughly and encourage fuzzer developers to expose more configuration parameters pertaining to these design decisions to end users.
DOI 原文 ·
@inproceedings{paperbot8453,
title = {An Empirical Examination of Fuzzer Mutator Performance},
author = {James Kukucka and Luís Pina and Paul Ammann and Jonathan Bell},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3680387}
}