一句话：首个ML流水线配置问题实证研究，提出Piecer自动生成库版本组合变体并检测性能不一致。
机器学习流水线通过 ML 库 API 组织学习任务，但其性能会随库版本及版本组合显著变化。配置不当的流水线可能导致执行时间/内存变差、数值错误甚至崩溃；若版本变化引起性能显著不一致，则视为流水线配置（PLC）问题。此前缺乏对其普遍性、影响和根因的系统研究。本文首次开展 PLC 问题的实证研究，并提出 Piecer：自动改变 ML 库版本组合生成一组流水线变体，并比较其性能不一致性。作者从 Kaggle 多个竞赛收集 11,363 条 ML 流水线，对其中 3,380 条可部署流水线应用 Piecer，发现 1,092 条存在 PLC 问题。
A Machine Learning (ML) pipeline configures the workflow of a learning task using the APIs provided by ML libraries. However, a pipeline’s performance can vary significantly across different configurations of ML library versions. Misconfigured pipelines can result in inferior performance, such as poor execution time and memory usage, numeric errors and even crashes. A pipeline is subject to misconfiguration if it exhibits significantly inconsistent performance upon changes in the versions of its configured libraries or the combination of these libraries. We refer to such performance inconsistency as a pipeline configuration (PLC) issue. There is no prior systematic study on the pervasiveness, impact and root causes of PLC issues. A systematic understanding of these issues helps configure effective ML pipelines and identify misconfigured ones. In this paper, we conduct the first empirical study of PLC issues. To better dig into the problem, we propose Piecer, an infrastructure that automatically generates a set of pipeline variants by varying different version combinations of ML libraries and compares their performance inconsistencies. We apply Piecer to the 3,380 pipelines that can be deployed out of the 11,363 ML pipelines collected from multiple ML competitions at Kaggle platform. The empirical study results show that 1,092 (32.3
@inproceedings{WangWZ0C0023,
title = {Can Machine Learning Pipelines Be Better Configured?},
author = {Yibo Wang and Ying Wang and Tingwei Zhang and Yue Yu and Shing-Chi Cheung and Hai Yu and Zhiliang Zhu},
booktitle = {FSE},
year = {2023},
doi = {10.1145/3611643.3616352}
}