一句话：大规模实证研究显示 Hugging Face 模型卡示例代码 65.98% 无法执行，并归纳出三类根因。
示例代码是用户理解与采用 Hugging Face 预训练模型的重要入口，但实际执行时频繁报错，损害可复现性与可用性，而此前缺乏系统评估。本文对模型卡示例代码的可执行性开展大规模实证研究：收集并执行 291 个模型卡的示例代码，结合执行日志分析失败模式与根因。结果显示，65.98% 的示例代码无法执行；失败率随模型规模增大而上升，且在语音、视觉、多模态与生成任务上尤为脆弱。根因主要有三类：依赖与版本要求文档不足、示例代码中隐含的运行时环境假设，以及生态演进（如 API 变更）导致的不兼容。该研究为评估模型卡示例在实际使用场景中的有效性提供了初步实证依据。
Example code in Hugging Face model cards serves as an important entry point for users to understand and adopt pretrained models; however, during execution, users frequently encounter execution errors, which in turn undermines reproducibility and practical usability. Given the long-standing lack of systematic evaluation of this phenomenon, we conduct a large-scale empirical study on the executability of model-card example code. Our results show that, among 291 model cards, 65.98% of example code fails to execute, indicating substantial limitations in real-world usability. Further analysis reveals that failure rates increase with model size and vary significantly across task types, with speech, vision, multimodal, and generative tasks being particularly fragile. Based on execution logs, we identify three dominant root causes: insufficient documentation of dependencies and version requirements, implicit runtime environment assumptions embedded in example code, and incompatibilities induced by ecosystem evolution (e.g., API changes). This study provides preliminary empirical evidence for assessing the effectiveness of model-card examples in real-world usage scenarios.
@inproceedings{paperbot11897,
title = {Understanding the Executability of Example Code in Hugging Face Model Cards: An Empirical Study},
author = {Simeng Yao and Jialin Zhao and Yang Zhang and Yiwen Wu and Tao Wang and Xunhui Zhang},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3774748.3795671}
}