一句话：将深度分类器训练动态的时间旅行可视化器发展为交互式调试器，通过人机回环推荐异常样本以辅助模型调试。
深度分类器的训练动态经时间旅行可视化后呈现为动画，但海量移动点与数百训练帧使开发者难以捕捉重要事件。本文首次将模型时间旅行可视化器发展为调试器，提出 DeepDebugger：给定样本表示与分类边界的训练动态动画，以人机回环方式推荐用户感兴趣的样本。它监控样本训练动态，按异常程度推荐可疑样本，并通过用户反馈迭代细化推荐，且对错误反馈具有容错性。在已知时间旅行可视化器上的实验表明，它能检测大多数异常样本运动，推荐性能较基线提升 5–10 倍，每次反馈开销仅 0.015 秒，在 3%、5%、10% 错误反馈下保持韧性；用户研究显示其节省 18.1% 完成时间并提升 20.3% 性能。
A deep classifier is usually trained to (i) learn the numeric representation vector of samples and (ii) classify sample representations with learned classification boundaries. Time-travelling visualization, as an explainable AI technique, is designed to transform the model training dynamics into an animation of canvas with colorful dots and territories. Despite that the training dynamics of the high-level concepts such as sample representations and classification boundaries are now observable, the model developers can still be overwhelmed by tens of thousands of moving dots across hundreds of training epochs (i.e., frames in the animation), which makes them miss important training events. In this work, we make the first attempt to develop the model time-travelling visualizers to the model time-travelling debuggers, for its practical use in model debugging tasks. Specifically, given an animation of model training dynamics of sample representation and classification landscape, we propose DeepDebugger solution to recommend the samples of user interest in a human-in-the-loop manner. On one hand, DeepDebugger monitors the training dynamics of samples and recommends suspicious samples based on their abnormality. On the other hand, our recommendation is interactive and fault-resilient for the model developers to explore the training process. By learning users’ feedback, DeepDebugger refines its recommendation to fit their intention. Our extensive experiments on applying DeepDebugger on the known time-travelling visualizers show that DeepDebugger can (1) detect the majority of the abnormal movement of the training samples on canvas; (2) significantly boost the recommendation performance of samples of interest (5-10X more accurate than the baselines) with the runtime overhead of 0.015s per feedback; (3) be resilient under the 3%, 5%, 10% mistaken user feedback. Our user study of the tool shows that the interactive recommendation of DeepDebugger can help the participants accomplish the debugging tasks by saving 18.1% completion time and boosting the performance by 20.3%.
@inproceedings{Yang0ZHD023,
title = {DeepDebugger: An Interactive Time-Travelling Debugging Approach for Deep Classifiers},
author = {Xianglin Yang and Yun Lin and Yifan Zhang and Linpeng Huang and Jin Song Dong and Hong Mei},
booktitle = {FSE},
year = {2023},
doi = {10.1145/3611643.3616252}
}