一句话：提出基于图的模糊测试方法，用MCTS驱动DL模型变异生成，提升推理引擎算子级覆盖率并发现大量异常。
现有深度学习测试技术多关注模型本身的质量，而忽视了底层推理引擎（框架与库）的可靠性。本文受模糊测试成功经验启发，提出一种契合深度学习模型图结构的基于图的模糊测试方法。工作引入了一种基于图论的算子级覆盖率准则，并实现了六种不同的变异策略，通过探索模型结构、参数与数据输入的组合来生成多样化的DL模型；同时采用蒙特卡洛树搜索（MCTS）在无需训练过程的情况下驱动模型生成。实验表明，MCTS在提升算子级覆盖率和检测异常方面均优于随机方法。该方法在模型转换失败、推理失败、输出比较失败三类不良行为中发现了40余种不同异常；变异策略可有效生成新的合法测试输入，平均提升算子级覆盖率最高达8.2%，并多捕获8.6个异常。
With the wide use of Deep Learning (DL) systems, academy and industry begin to pay attention to their quality. Testing is one of the major methods of quality assurance. However, existing testing techniques focus on the quality of DL models but lacks attention to the core underlying inference engines (i.e., frameworks and libraries). Inspired by the success stories of fuzz testing, we design a graph-based fuzz testing method to improve the quality of DL inference engines. This method is naturally followed by the graph structure of DL models. A novel operator-level coverage criterion based on graph theory is introduced and six different mutations are implemented to generate diversified DL models by exploring combinations of model structures, parameters, and data inputs. The Monte Carlo Tree Search (MCTS) is used to drive DL model generation without a training process. The experimental results show that the MCTS outperforms the random method in boosting operator-level coverage and detecting exceptions. Our method has discovered more than 40 different exceptions in three types of undesired behaviors: model conversion failure, inference failure, output comparison failure. The mutation strategies are useful to generate new valid test inputs, by up to an 8.2% more operator-level coverage on average and 8.6 more exceptions captured.
@inproceedings{abs-2008-05933,
title = {Graph-Based Fuzz Testing for Deep Learning Inference Engines},
author = {Weisi Luo and Dong Chai and Xiaoyue Ruan and Jiang Wang and Chunrong Fang and Zhenyu Chen},
booktitle = {ICSE},
year = {2021},
doi = {10.1109/icse43902.2021.00037}
}