In recent years, coverage-based greybox fuzzing has proven to be an effective and practical technique for discovering software vulnerabilities. The availability of American Fuzzy Loop (AFL) has facilitated numerous advances in overcoming challenges in fuzzing. However, the issue of mutating complex file formats, such as PDF, remains unresolved due to strict constraints. Existing fuzzers often produce mutants that fail to parse by applications, limited by bit/byte mutations performed on input files. Our observation is that most in-memory representations of file formats are simple, and well-designed applications have built-in printer functions to emit these structures as files. Thus, we propose a new technique that mutates the in-memory structures of inputs and utilizes printer functions to regenerate mutated files. Unlike prior approaches that require complex analysis to learn file format constraints, our technique leverages the printer function to preserve format constraints. We implement a prototype called FuzzInMem and compare it with AFL as well as other state-of-the-art fuzzers, including AFL++, Mopt, Weizz, and FormatFuzzer. The results show that FuzzInMem is scalable and substantially outperforms general-purpose fuzzers in terms of valid seed generation and path coverage. By applying FuzzInMem to real-world applications, we found 29 unique vulnerabilities and were awarded 5 CVEs.

@inproceedings{paperbot11442,
  title = {FuzzInMem: Fuzzing Programs via In-memory Structures},
  author = {Xuwei Liu and Wei You and Yapeng Ye and Zhuo Zhang and Jianjun Huang and Xiangyu Zhang},
  booktitle = {ICSE},
  year = {2024},
  doi = {10.1145/3597503.3639172}
}