一句话：通过 92 名工程师问卷与 200 个 ONNX 缺陷分析，揭示 DL 模型转换器约 75% 的缺陷集中在节点转换阶段。
深度学习模型转换器负责在开发框架与运行时环境之间迁移模型，转换错误会损害模型质量并阻碍部署，但其失败特征此前并不清楚。本文对 ONNX 模型转换器的失败展开研究：首先对 92 名软件工程师进行问卷调查，了解 DL 互操作工具的使用场景与痛点；其次对 PyTorch 与 TensorFlow 生态中 200 个 ONNX 相关 issue 进行失败刻画；最后提出并检验两个关于失败结构性成因的假设。结果显示，转换器的节点转换阶段约占缺陷的 75%，33% 的报告失败与语义不正确的模型有关；语义错误的根因难以定位，但行为不一致的模型往往共享算子序列。作者据此呼吁让 DL 互操作软件更易维护、扩展与验证，并指出行为容差与架构覆盖度度量是未来值得研究的方向。
Software engineers develop, fine-tune, and deploy deep learning (DL) models using a variety of development frameworks and runtime environments. DL model converters move models between frameworks and to runtime environments. Conversion errors compromise model quality and disrupt deployment. However, the failure characteristics of DL model converters are unknown, adding risk when using DL interoperability technologies. This paper analyzes failures in DL model converters. We survey software engineers about DL interoperability tools, use cases, and pain points (N=92). Then, we characterize failures in model converters associated with the main interoperability tool, ONNX (N=200 issues in PyTorch and TensorFlow). Finally, we formulate and test two hypotheses about structural causes for the failures we studied. We find that the node conversion stage of a model converter accounts for ∼75% of the defects and 33% of reported failure are related to semantically incorrect models. The cause of semantically incorrect models is elusive, but models with behaviour inconsistencies share operator sequences. Our results motivate future research on making DL interoperability software simpler to maintain, extend, and validate. Research into behavioural tolerances and architectural coverage metrics would be fruitful.
@inproceedings{paperbot8557,
title = {Interoperability in Deep Learning: A User Survey and Failure Analysis of ONNX Model Converters},
author = {Purvish Jajal and Wenxin Jiang and Arav Tewari and Erik Kocinare and Joseph Woo and Anusha Sarraf and Yung-Hsiang Lu and George K. Thiruvathukal and James C. Davis},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3680374}
}