一句话：提出 DevMuT，利用开发者经验驱动的变异算子和约束测试深度学习框架，发现更受开发者关注的重要缺陷。
深度学习框架缺陷可能导致严重后果，但现有基于模型变异测试的方法所发现的缺陷多被开发者视为边角案例或忽略。DevMuT 从开发者专业经验中提取变异算子与约束来生成测试模型，模拟开发中的常见操作，从而覆盖模型训练、推理等更多生命周期阶段并暴露更多样缺陷。作者在 PyTorch、JAX 和 MindSpore 三个框架、来自九类行业任务的 29 个模型上评估。结果显示，生成模型的多样性平均至少提升 71.68%，合法率平均提升 28.20%；共发现 117 个缺陷，其中 63 个确认、24 个修复、8 个被开发者确认为高价值，并自 2023 年 12 月起部署于 MindSpore 社区。
Deep learning (DL) frameworks are the fundamental infrastructure for various DL applications. Framework defects can profoundly cause disastrous accidents, thus requiring sufficient detection. In previous studies, researchers adopt DL models as test inputs combined with mutation to generate more diverse models. Though these studies demonstrate promising results, most detected defects are considered trivial (i.e., either treated as edge cases or ignored by the developers). To identify important bugs that matter to developers, we propose a novel DL framework testing method DevMuT, which generates models by adopting mutation operators and constraints derived from developer expertise. DevMuT simulates developers' common operations in development and detects more diverse defects within more stages of the DL model lifecycle (e.g., model training and inference). We evaluate the performance of DevMuT on three widely used DL frameworks (i.e., PyTorch, JAX, and Mind-Spore) with 29 DL models from nine types of industry tasks. The experiment results show that DevMuT outperforms state-of-the-art baselines: it can achieve at least 71.68% improvement on average in the diversity of generated models and 28.20% improvement on average in the legal rates of generated models. Moreover, DevMuT detects 117 defects, 63 of which are confirmed, 24 are fixed, and eight are of high value confirmed by developers. Finally, DevMuT has been deployed in the MindSpore community since December 2023. These demonstrate the effectiveness of DevMuT in detecting defects that are close to the real scenes and are of concern to developers.
DOI 原文 ·
@inproceedings{paperbot8429,
title = {DevMuT: Testing Deep Learning Framework via Developer Expertise-Based Mutation},
author = {Yanzhou Mu and Juan Zhai and Chunrong Fang and Xiang Chen and Zhixiang Cao and Peiran Yang and Yinglong Zou and Tao Zheng and Zhenyu Chen},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695523}
}