一句话：对部署中 DNN 可靠性评估方法进行复现与比较，发现难以复现和比较，并提出统一评估框架。
随着深度神经网络进入安全关键应用，其可靠性备受关注；即使训练充分且性能很高的网络也并非完全准确，而部署时缺乏真实标签使行为预测十分困难。本文对近期提出的部署中 DNN 可靠性评估方法开展比较与可复现性研究：在作者提供的复现包上运行，并尝试迁移到其他工件，同时分析其评估指标。结果显示这些方法难以运行和复现结果，迁移到非自身工件更加困难，且因缺乏明确定义的指标而难以公平比较有效性。作者据此指出社区需进一步努力，并贡献一个集成已有方法的评估框架，以公共基准和通用指标进行评价。
As deep neural networks (DNNs) are increasingly used in safety-critical applications, there is a growing concern for their reliability. Even highly trained, high-performant networks are not 100% accurate. However, it is very difficult to predict their behavior during deployment without ground truth. In this paper, we provide a comparative and replicability study on recent approaches that have been proposed to evaluate the reliability of DNNs in deployment. We find that it is hard to run and reproduce the results for these approaches on their replication packages and even more difficult to run them on artifacts other than their own. Further, it is difficult to compare the effectiveness of the approaches, due to the lack of clearly defined evaluation metrics. Our results indicate that more effort is needed in our research community to obtain sound techniques for evaluating the reliability of neural networks in safety-critical domains. To this end, we contribute an evaluation framework that incorporates the considered approaches and enables evaluation on common benchmarks, using common metrics.
@inproceedings{paperbot8577,
title = {Evaluating Deep Neural Networks in Deployment: A Comparative Study (Replicability Study)},
author = {Eduard Pinconschi and Divya Gopinath and Rui Abreu and Corina S. Păsăreanu},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3680401}
}