一句话：提出 DeepREL，自动推断深度学习库中的关系 API，并据此合成测试与模糊测试，显著提升 API 覆盖并发现大量缺陷。
深度学习库（如 PyTorch、TensorFlow）是构建与训练模型的基础，但其自身缺陷可能带来严重后果。现有模糊测试工作只能为已被文档示例、开发者测试或模型调用过的 API 生成测试，导致大量 API 未被覆盖。本文提出 DeepREL，首个自动推断关系 API 以增强深度学习库模糊测试的方法。其核心假设是：被测库中存在许多共享相似输入参数与输出的 API，因此可从已调用 API“借用”测试输入来测试其他关系 API。作者形式化了关系 API 的值等价与状态等价，并将其作为发现缺陷的 oracle。DeepREL 端到端自动完成 API 关系推断、测试程序合成、关系 API 验证与模糊测试。在 PyTorch 和 TensorFlow 上的评估显示，其 API 覆盖比 FreeFuzz 高 157%，共检测 162 个代码缺陷，其中 106 个已被开发者确认为未知缺陷；三个月内检测到 PyTorch 整个问题追踪系统中 13.5% 的高优先级缺陷，另发现 14 个文档缺陷且全部确认。
Deep Learning (DL) has gained wide attention in recent years. Meanwhile, bugs in DL systems can lead to serious consequences, and may even threaten human lives. As a result, a growing body of research has been dedicated to DL model testing. However, there is still limited work on testing DL libraries, e.g., PyTorch and TensorFlow, which serve as the foundations for building, training, and running DL models. Prior work on fuzzing DL libraries can only generate tests for APIs which have been invoked by documentation examples, developer tests, or DL models, leaving a large number of APIs untested. In this paper, we propose DeepREL, the first approach to automatically inferring relational APIs for more effective DL library fuzzing. Our basic hypothesis is that for a DL library under test, there may exist a number of APIs sharing similar input parameters and outputs; in this way, we can easily “borrow” test inputs from invoked APIs to test other relational APIs. Furthermore, we formalize the notion of value equivalence and status equivalence for relational APIs to serve as the oracle for effective bug finding. We have implemented DeepREL as a fully automated end-to-end relational API inference and fuzzing technique for DL libraries, which 1) automatically infers potential API relations based on API syntactic/semantic information, 2) synthesizes concrete test programs for invoking relational APIs, 3) validates the inferred relational APIs via representative test inputs, and finally 4) performs fuzzing on the verified relational APIs to find potential inconsistencies. Our evaluation on two of the most popular DL libraries, PyTorch and TensorFlow, demonstrates that DeepREL can cover 157% more APIs than state-of-the-art FreeFuzz. To date, DeepREL has detected 162 bugs in total, with 106 already confirmed by the developers as previously unknown bugs. Surprisingly, DeepREL has detected 13.5% of the high-priority bugs for the entire PyTorch issue-tracking system in a three-month period. Also, besides the 162 code bugs, we have also detected 14 documentation bugs (all confirmed).
DOI 原文 ·
@inproceedings{paperbot6994,
title = {Fuzzing deep-learning libraries via automated relational API inference},
author = {Yinlin Deng and Chenyuan Yang and Anjiang Wei and Lingming Zhang},
booktitle = {FSE},
year = {2022},
doi = {10.1145/3540250.3549085}
}