一句话：测量四个模型库中36个预训练DNN在准确率、延迟和架构上的差异，揭示模型库可靠性威胁。
预训练深度神经网络（PTNN）可加速部署，但模型库（model zoo）的可靠性尚未得到检验，且缺乏实现与性能的行业标准，工程师难以放心将其用于生产系统。本文作为第一步，量化四个模型库中36个PTNN在准确率、延迟和架构上的显著差异。研究发现，排名前十的差异中准确率相差1.23%–2.62%，延迟相差9%–131%；对于ResNet、AlexNet等知名DNN架构也存在架构不匹配。这些差异揭示了模型库可靠性的潜在威胁，作者呼吁开展实证验证、自动测量工具以及实现最佳实践。
Training deep neural networks (DNNs) takes significant time and resources. A practice for expedited deployment is to use pre-trained deep neural networks (PTNNs), often from model zoos--collections of PTNNs; yet, the reliability of model zoos remains unexamined. In the absence of an industry standard for the implementation and performance of PTNNs, engineers cannot confidently incorporate them into production systems. As a first step, discovering potential discrepancies between PTNNs across model zoos would reveal a threat to model zoo reliability. Prior works indicated existing variances in deep learning systems in terms of accuracy. However, broader measures of reliability for PTNNs from model zoos are unexplored. This work measures notable discrepancies between accuracy, latency, and architecture of 36 PTNNs across four model zoos. Among the top 10 discrepancies, we find differences of 1.23%-2.62% in accuracy and 9%-131% in latency. We also find mismatches in architecture for well-known DNN architectures (e.g., ResNet and AlexNet). Our findings call for future works on empirical validation, automated tools for measurement, and best practices for implementation.
@inproceedings{paperbot7109,
title = {Discrepancies among pre-trained deep neural networks: a new threat to model zoo reliability},
author = {Diego Montes and Pongpatapee Peerapatanapokin and Jeff Schultz and Chengjun Guo and Wenxin Jiang and James C. Davis},
booktitle = {FSE},
year = {2022},
doi = {10.1145/3540250.3560881}
}