一句话：提出QAQA，基于句子级变异蜕变测试，用语义引导搜索和增强预言实现问答软件精确测试。
问答软件基于信息检索与自然语言处理自动回答自然语言问题，但可能含缺陷。现有无标注测试从问答对抽取事实并生成新问题，然而生成的问题可能歧义、混乱或语法异常，导致问答软件无法回答，造成较多误报。本文提出QAQA，一种句子级变异的蜕变测试技术：它使用五个蜕变关系，结合语义引导搜索与增强测试预言，以消除误报并实现精确自动测试。在三个问答数据集上的评估表明，QAQA在报告缺陷数量（8133对6601）和质量（真阳性率97.67%对49%）上均优于最新方法；其测试输入用于微调被测软件时，还将蜕变关系违反率从44.29%降至20.51%。
Question answering (QA) software uses information retrieval and natural language processing techniques to automatically answer questions posed by humans in a natural language. Like other AI-based software, QA software may contain bugs. To automatically test QA software without human labeling, previous work extracts facts from question answer pairs and generates new questions to detect QA software bugs. Nevertheless, the generated questions could be ambiguous, confusing, or with chaotic syntax, which are unanswerable for QA software. As a result, a relatively large proportion of the reported bugs are false positives. In this work, we proposed QAQA, a sentence-level mutation based metamorphic testing technique for QA software. To eliminate false positives and achieve precise automatic testing, QAQA leverages five Metamorphic Relations (MRs) as well as semantics-guided search and enhanced test oracles. Our evaluation on three QA datasets demonstrates that QAQA outperforms the state-of-the-art in both quantity (8,133 vs. 6,601 bugs) and quality (97.67% vs. 49% true positive rate) of the reported bugs. Moreover, the test inputs generated by QAQA successfully reduce MR violation rate from 44.29% to 20.51% when being adopted in fine-tuning the QA software under test.
@inproceedings{paperbot7190,
title = {Natural Test Generation for Precise Testing of Question Answering Software},
author = {Qingchao Shen and Junjie Chen and Jie M. Zhang and Haoyu Wang and Shuang Liu and Menghan Tian},
booktitle = {ASE},
year = {2022},
doi = {10.1145/3551349.3556953}
}