一句话：首个针对机器学习模型优化缺陷（MOBs）的大规模实证研究，基于 TensorFlow 与 PyTorch 的 371 个缺陷刻画其症状、根因与修复。
为适配移动与 IoT 等资源受限平台，预训练模型常借助剪枝、量化等优化技术提升效率与紧凑性；与编译器、数据库等复杂系统中的优化类似，ML 模型优化也可能含缺陷，导致系统崩溃与经济损失。尽管训练、编译与部署阶段的缺陷已被广泛研究，模型优化缺陷（MOBs）仍缺乏系统认识。本文开展首个实证研究：从 TensorFlow 与 PyTorch 两个主流开源 ML 框架中收集覆盖其优化器完整开发时间跨度（2019 年 5 月至 2022 年 8 月）的 371 个 MOBs，并从症状、根因、生命周期、检测与修复等多视角进行分析。研究揭示了 MOBs 的真实状况及其可用于未来检测技术的特征，警示框架开发者与用户，并呼吁社区制定针对性对策。
Due to the great advance in machine learning (ML) techniques, numerous ML models are expanding their application domains in recent years. To adapt for resource-constrained platforms such as mobile and Internet of Things (IoT) devices, pre-trained models are often processed to enhance their efficiency and compactness, using optimization techniques such as pruning and quantization. Similar to the optimization process in other complex systems, e.g., program compilers and databases, optimizations for ML models can contain bugs, leading to severe consequences such as system crashes and financial loss. While bugs in training, compiling and deployment stages have been extensively studied, there is still a lack of systematic understanding and characterization of model optimization bugs (MOBs). In this work, we conduct the first empirical study to identify and characterize MOBs. We collect a comprehensive dataset containing 371 MOBs from TensorFlow and PyTorch, the most extensively used open-source ML frameworks, covering the entire development time span of their optimizers (May 2019 to August 2022). We then investigate the collected bugs from various perspectives, including their symptoms, root causes, life cycles, detection and fixes. Our work unveils the status quo of MOBs in the wild, and reveals their features on which future detection techniques can be based. Our findings also serve as a warning to the developers and the users of ML frameworks, and an appeal to our research community to enact dedicated countermeasures.
@inproceedings{GuanXLLB23,
title = {A Comprehensive Study of Real-World Bugs in Machine Learning Model Optimization},
author = {Hao Guan and Ying Xiao and Jiaying Li and Yepang Liu and Guangdong Bai},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00024}
}