一句话：首次实证研究 Hugging Face 中预训练模型复用实践，建模决策过程并量化其挑战。
随着 DNN 成为软件系统组件，从头构建和特化模型愈发困难，工程师开始复用并微调大规模预训练模型（PTM），但缺乏 PTM 生态的复用知识。本文首次对 PTM 复用进行实证研究：访谈 Hugging Face 的 12 名实践者，归纳复用实践与挑战，并建立 PTM 复用决策过程模型。基于此，作者提出模型复用的有用属性，包括来源、可复现性与可移植性；指出三类挑战：属性缺失、宣称性能与实际性能不一致以及模型风险。随后在 Hugging Face 生态中进行系统测量以佐证这些挑战。工作为通过自动度量有用属性与潜在攻击、以及模型注册表基础设施和标准化研究提供了方向。
Deep Neural Networks (DNNs) are being adopted as components in software systems. Creating and specializing DNNs from scratch has grown increasingly difficult as state-of-the-art architectures grow more complex. Following the path of traditional software engineering, machine learning engineers have begun to reuse large-scale pre-trained models (PTMs) and fine-tune these models for downstream tasks. Prior works have studied reuse practices for traditional software packages to guide software engineers towards better package maintenance and dependency management. We lack a similar foundation of knowledge to guide behaviors in pre-trained model ecosystems. In this work, we present the first empirical investigation of PTM reuse. We interviewed 12 practitioners from the most popular PTM ecosystem, Hugging Face, to learn the practices and challenges of PTM reuse. From this data, we model the decision-making process for PTM reuse. Based on the identified practices, we describe useful attributes for model reuse, including provenance, reproducibility, and portability. Three challenges for PTM reuse are missing attributes, discrepancies between claimed and actual performance, and model risks. We substantiate these identified challenges with systematic measurements in the Hugging Face ecosystem. Our work informs future directions on optimizing deep learning ecosystems by automated measuring useful attributes and potential attacks, and envision future research on infrastructure and standardization for model registries.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2303-02552,
title = {An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry},
author = {Wenxin Jiang and Nicholas Synovic and Matt Hyatt and Taylor R. Schorlemmer and Rohan Sethi and Yung-Hsiang Lu and George K. Thiruvathukal and James C. Davis},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00206}
}