一句话：面向工业真空抽气的样本稀缺场景，提出基于物理原理的数据增强与鲁棒性测试方法，提升ML模型可靠性。
工业制造中的机器学习分类与回归常面临训练和测试样本稀缺，尤其电渣重熔真空抽气过程每年仅产生数百个样本，少量测试数据不足以评估模型鲁棒性。本文基于真空抽气原理生成增强样本，并设计三种测试场景与一个测试 oracle，用于评估生产级ML模型的鲁棒性。实验使用Uddeholms AB钢铁公司的真实工业数据，结果显示Ensemble与神经网络在增强数据训练下最鲁棒，所提测试策略能有效检查并提升模型鲁棒性。最后给出面向边缘节点实时预测和云端持续交付的MLOps实现。
There is often a scarcity of training data for machine learning (ML) classification and regression models in industrial production, especially for time-consuming or sparsely run manufacturing processes. Traditionally, a majority of the limited ground-truth data is used for training, while a handful of samples are left for testing. In that case, the number of test samples is inadequate to properly evaluate the robustness of the ML models under test (i.e., the system under test) for classification and regression. Furthermore, the output of these ML models may be inaccurate or even fail if the input data differ from the expected. This is the case for ML models used in the Electroslag Remelting (ESR) process in the refined steel industry to predict the pressure in a vacuum chamber. A vacuum pumping event that occurs once a workday generates a few hundred samples in a year of pumping for training and testing. In the absence of adequate training and test samples, this paper first presents a method to generate a fresh set of augmented samples based on vacuum pumping principles. Based on the generated augmented samples, three test scenarios and one test oracle are presented to assess the robustness of an ML model used for production on an industrial scale. Experiments are conducted with real industrial production data obtained from Uddeholms AB steel company. The evaluations indicate that Ensemble and Neural Network are the most robust when trained on augmented data using the proposed testing strategy. The evaluation also demonstrates the proposed method's effectiveness in checking and improving ML algorithms' robustness in such situations. The work improves software testing's state-of-the-art robustness testing in similar settings. Finally, the paper presents an MLOps implementation of the proposed approach for real-time ML model prediction and action on the edge node and automated continuous delivery of ML software from the cloud.
@inproceedings{paperbot7008,
title = {Testing of machine learning models with limited samples: an industrial vacuum pumping application},
author = {Ayan Chatterjee and Bestoun S. Ahmed and Erik Hallin and Anton Engman},
booktitle = {FSE},
year = {2022},
doi = {10.1145/3540250.3558943}
}