一句话：通过Surprise Adequacy与模型性能的相关性预测未标注输入的性能，从而降低自动驾驶DNN分割的标注成本。
自动驾驶中的目标分割DNN是安全关键任务，其开发需反复进行数据收集、标注、训练与评估，其中数据标注是昂贵的人工瓶颈。本文利用Surprise Adequacy（SA）与模型性能之间的相关性，在不进行人工标注的情况下预测模型对输入的表現，从而帮助理解模型性能、指导数据收集并支持后续训练决策。作者在工业案例中验证该方法，最高可节省50%的标注成本，且评估误差可忽略；同时，工程师可根据不同开发阶段和场景，在成本节省与可接受误差之间进行权衡。
Deep Neural Networks (DNNs) are rapidly being adopted by the automotive industry, due to their impressive performance in tasks that are essential for autonomous driving. Object segmentation is one such task: its aim is to precisely locate boundaries of objects and classify the identified objects, helping autonomous cars to recognise the road environment and the traffic situation. Not only is this task safety critical, but developing a DNN based object segmentation module presents a set of challenges that are significantly different from traditional development of safety critical software. The development process in use consists of multiple iterations of data collection, labelling, training, and evaluation. Among these stages, training and evaluation are computation intensive while data collection and labelling are manual labour intensive. This paper shows how development of DNN based object segmentation can be improved by exploiting the correlation between Surprise Adequacy (SA) and model performance. The correlation allows us to predict model performance for inputs without manually labelling them. This, in turn, enables understanding of model performance, more guided data collection, and informed decisions about further training. In our industrial case study the technique allows cost savings of up to 50% with negligible evaluation inaccuracy. Furthermore, engineers can trade off cost savings versus the tolerable level of inaccuracy depending on different development phases and scenarios.
@inproceedings{paperbot5712,
title = {Reducing DNN labelling cost using surprise adequacy: an industrial case study for autonomous driving},
author = {Jinhan Kim and Jeongil Ju and Robert Feldt and Shin Yoo},
booktitle = {FSE},
year = {2020},
doi = {10.1145/3368089.3417065}
}