一句话：提出MODE，通过模型状态差异分析与训练输入选择自动调试并修复神经网络模型缺陷。
论文研究AI模型缺陷导致精度下降且难以直接修改模型修复的问题。现有方法靠额外训练输入修复，但因不理解模型异常、无法选好输入而效果有限。MODE受软件调试启发，先通过模型状态差异分析定位造成缺陷的内部特征，再借鉴回归测试的程序输入选择筛选训练输入。在6类应用、29个模型上评估，MODE能有效高效修复缺陷且不引入新缺陷：数字识别平均测试精度从75%提至93%，现有最优仅85%且需约11倍训练时间；物体识别等复杂模型可从75%提至91%以上，耗时数分钟到数小时，而现有方法失败甚至降低精度。
Artificial intelligence models are becoming an integral part of modern computing systems. Just like software inevitably has bugs, models have bugs too, leading to poor classification/prediction accuracy. Unlike software bugs, model bugs cannot be easily fixed by directly modifying models. Existing solutions work by providing additional training inputs. However, they have limited effectiveness due to the lack of understanding of model misbehaviors and hence the incapability of selecting proper inputs. Inspired by software debugging, we propose a novel model debugging technique that works by first conducting model state differential analysis to identify the internal features of the model that are responsible for model bugs and then performing training input selection that is similar to program input selection in regression testing. Our evaluation results on 29 different models for 6 different applications show that our technique can fix model bugs effectively and efficiently without introducing new bugs. For simple applications (e.g., digit recognition), MODE improves the test accuracy from 75% to 93% on average whereas the state-of-the-art can only improve to 85% with 11 times more training time. For complex applications and models (e.g., object recognition), MODE is able to improve the accuracy from 75% to over 91% in minutes to a few hours, whereas state-of-the-art fails to fix the bug or even degrades the test accuracy.
DOI 原文 ·
@inproceedings{paperbot4604,
title = {MODE: automated neural network model debugging via state differential analysis and input selection},
author = {Shiqing Ma and Yingqi Liu and Wen-Chuan Lee and Xiangyu Zhang and Ananth Grama},
booktitle = {FSE},
year = {2018},
doi = {10.1145/3236024.3236082}
}