一句话：提出基于蜕变测试的 ASRTest，为 DNN 驱动的语音识别系统生成测试语音并检测错误行为。
深度神经网络与端到端学习推动了自动语音识别（ASR）系统的广泛应用，但这类系统仍可能存在软件缺陷并表现出错误行为。由于 DNN 的特性，传统软件测试技术难以直接适用，同时测试数据多样性和判定预言（oracle）信息的缺乏进一步阻碍了 ASR 系统测试。本文提出并实现了 ASRTest，一种面向 DNN 驱动 ASR 系统的测试方法。ASRTest 基于蜕变测试理论，首先为 ASR 系统设计蜕变关系，然后实现三类变换算子以模拟实际应用场景并生成语音；同时采用 Gini 不纯度引导生成过程以提高测试效率。作者在四个 ASR 模型和四个常用数据集上评估 ASRTest，结果表明其能在不同真实应用条件下高效检测错误行为，并通过使用生成数据重训练平均提升 19.1% 的识别性能。此外，对一个工业 ASR 系统的案例研究表明，ASRTest 能有效检测错误并提升 DNN 驱动 ASR 系统的性能。
With the rapid development of deep neural networks and end-to-end learning techniques, automatic speech recognition (ASR) systems have been deployed into our daily and assist in various tasks. However, despite their tremendous progress, ASR systems could also suffer from software defects and exhibit incorrect behaviors. While the nature of DNN makes conventional software testing techniques inapplicable for ASR systems, lacking diverse tests and oracle information further hinders their testing. In this paper, we propose and implement a testing approach, namely ASR, specifically for the DNN-driven ASR systems. ASRTest is built upon the theory of metamorphic testing. We first design the metamorphic relation for ASR systems and then implement three families of transformation operators that can simulate practical application scenarios to generate speeches. Furthermore, we adopt Gini impurity to guide the generation process and improve the testing efficiency. To validate the effectiveness of ASRTest, we apply ASRTest to four ASR models with four widely-used datasets. The results show that ASRTest can detect erroneous behaviors under different realistic application conditions efficiently and improve 19.1% recognition performance on average via retraining with the generated data. Also, we conduct a case study on an industrial ASR system to investigate the performance of ASRTest under the real usage scenario. The study shows that ASRTest can detect errors and improve the performance of DNN-driven ASR systems effectively.
DOI 原文 ·
@inproceedings{paperbot7387,
title = {ASRTest: automated testing for deep-neural-network-driven speech recognition systems},
author = {Pin Ji and Yang Feng and Jia Liu and Zhihong Zhao and Zhenyu Chen},
booktitle = {ISSTA},
year = {2022},
doi = {10.1145/3533767.3534391}
}