一句话：提出 Muffin，利用模型模糊测试与训练阶段差分测试全面检测深度学习库缺陷，发现 39 个新 bug。
深度学习库是构建和执行深度学习模型的基础，但其中存在缺陷且可能导致严重后果。已有测试方法通常依赖现有已训练模型，仅能检测模型推理阶段的错误，难以全面覆盖库代码。Muffin 为此设计了模型模糊测试方法，可生成多样化的深度学习模型来探索目标库，而非局限于已有模型；同时通过定制一组度量指标，在模型训练阶段开展差分测试，衡量不同深度学习库之间的不一致性，从而更充分地执行库代码并发现更多缺陷。作者在 TensorFlow、CNTK 和 Theano 三个广泛使用的库上评估 Muffin，结果在最新发布版本中检测到 39 个新 bug。
Deep learning (DL) techniques are proven effective in many challenging tasks, and become widely-adopted in practice. However, previous work has shown that DL libraries, the basis of building and executing DL models, contain bugs and can cause severe consequences. Unfortunately, existing testing approaches still cannot comprehensively exercise DL libraries. They utilize existing trained models and only detect bugs in model inference phase. In this work we propose Muffin to address these issues. To this end, Muffin applies a specifically-designed model fuzzing approach, which allows it to generate diverse DL models to explore the target library, instead of relying only on existing trained models. Muffin makes differential testing feasible in the model training phase by tailoring a set of metrics to measure the inconsistencies between different DL libraries. In this way, Muffin can best exercise the library code to detect more bugs. To evaluate the effectiveness of Muffin, we conduct experiments on three widely-used DL libraries. The results demonstrate that Muffin can detect 39 new bugs in the latest release versions of popular DL libraries, including Tensorflow, CNTK, and Theano.
DOI 原文 ·
@inproceedings{paperbot6957,
title = {Muffin},
author = {Jiazhen Gu and Xuchuan Luo and Yangfan Zhou and Xin Wang},
booktitle = {ICSE},
year = {2022},
doi = {10.1145/3510003.3510092}
}