一句话：提出 ASTER，注入五类口吃生成测试语音，并用多目标优化种子更新，自动测试 ASR 对口吃者的可访问性。
ASR 系统对口吃语音的处理是其可访问性的重要方面，但现有口吃语音数据集多样性不足且缺乏非口吃文本真值，难以作为全面测试套件暴露失败。ASTER 面向该问题，通过注入五类口吃自动生成测试用例，使样本既模拟真实口吃语音又具备触发失败所需的多样性，并用基于多目标优化的种子更新算法提升测试质量。作者实现框架并在四个开源 ASR 模型和三个商业 ASR 系统上评估。结果显示，ASTER 显著提高词错误率、匹配错误率和词信息损失；用户研究也表明生成的口吃音频与真实口吃音频难以区分。
The popularity of automatic speech recognition (ASR) systems nowadays leads to an increasing need for improving their accessibility. Handling stuttering speech is an important feature for accessible ASR systems. To improve the accessibility of ASR systems for stutterers, we need to expose and analyze the failures of ASR systems on stuttering speech. The speech datasets recorded from stutterers are not diverse enough to expose most of the failures. Furthermore, these datasets lack ground truth information about the non-stuttered text, rendering them unsuitable as comprehensive test suites. Therefore, a methodology for generating stuttering speech as test inputs to test and analyze the performance of ASR systems is needed. However, generating valid test inputs in this scenario is challenging. The reason is that although the generated test inputs should mimic how stutterers speak, they should also be diverse enough to trigger more failures. To address the challenge, we propose Aster, a technique for automatically testing the accessibility of ASR systems. Aster can generate valid test cases by injecting five different types of stuttering. The generated test cases can both simulate realistic stuttering speech and expose failures in ASR systems. Moreover, Aster can further enhance the quality of the test cases with a multi-objective optimization-based seed updating algorithm. We implemented Aster as a framework and evaluated it on four open-source ASR models and three commercial ASR systems. We conduct a comprehensive evaluation of Aster and find that it significantly increases the word error rate, match error rate, and word information loss in the evaluated ASR systems. Additionally, our user study demonstrates that the generated stuttering audio is indistinguishable from real-world stuttering audio clips.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2308-15742,
title = {ASTER: Automatic Speech Recognition System Accessibility Testing for Stutterers},
author = {Yi Liu and Yuekang Li and Gelei Deng and Felix Juefei-Xu and Yao Du and Cen Zhang and Chengwei Liu and Yeting Li and Lei Ma and Yang Liu},
booktitle = {ASE},
year = {2023},
doi = {10.1109/ase56229.2023.00107}
}