一句话：提出 VUI-UPSET，将聊天机器人测试方法迁移到语音用户界面测试，生成更多能暴露缺陷的复述命令。
语音虚拟助手应用通过自然语言命令与用户交互，同一意图可有多种表达，导致 VUI 测试困难。现有开发环境集成的深度学习工具虽能为种子话语生成复述，但数量少且难以覆盖边界情况。本文提出 VUI-UPSET，借鉴聊天机器人测试方法来自动生成 VUI 复述。作者进行实证研究，比较 VUI-UPSET 与 Alexa Developer Console 中 DL 工具在复述正确性和缺陷揭示能力上的表现，人工分析了 5,872 条复述（共 13,310 次人工评估）。结果表明：DL 工具生成的有意义复述比例更高，而 VUI-UPSET 生成的复述更能暴露缺陷，代价是需丢弃更多无关复述，从而帮助开发者更彻底地测试应用。
Voice-based virtual assistants are becoming increasingly popular. Such systems provide frameworks to developers on which they can build their own apps. End-users can interact with such apps through a Voice User Interface (VUI), which allows to use natural language commands to perform actions. Testing such apps is far from trivial: The same command can be expressed in different ways. To support developers in testing VUIs, Deep Learning (DL)-based tools have been integrated in the development environments (e.g., the Alexa Developer Console, or ADC) to generate paraphrases for the commands (seed utterances) specified by the developers. Such tools, however, generate few paraphrases that do not always cover corner cases. In this paper, we introduce VUI-UPSET, a novel approach that aims at adapting chatbot-testing approaches to VUI-testing. Both systems, indeed, provide a similar natural-language-based interface to users. We conducted an empirical study to understand how VUI-UPSET compares to existing approaches in terms of (i) correctness of the generated paraphrases, and (ii) capability of revealing bugs. Multiple authors analyzed 5,872 generated paraphrases, with a total of 13,310 manual evaluations required for such a process. Our results show that, while the DL-based tool integrated in the ADC generates a higher percentage of meaningful paraphrases compared to VUI-UPSET, VUI-UPSET generates more bug-revealing paraphrases. This allows developers to test more thoroughly their apps at the cost of discarding a higher number of irrelevant paraphrases.
@inproceedings{paperbot7176,
title = {Sorry, I don’t Understand: Improving Voice User Interface Testing},
author = {Emanuela Guglielmi and Giovanni Rosa and Simone Scalabrino and Gabriele Bavota and Rocco Oliveto},
booktitle = {ASE},
year = {2022},
doi = {10.1145/3551349.3556934}
}