一句话：提出 qaAskeR，利用三种蜕变关系递归提问测试问答软件，无需标注标签即可揭示违规。
问答（QA）是自然语言处理中的重要且富有挑战的方向，已有多种算法和基准数据集，QA 软件也广泛用于日常生活。然而，当前 QA 软件主要采用基于参考的测试范式，需要先投入大量人工标注测试用例的预期输出。这导致既无法在使用过程中进行即时测试，也无法在大规模无标注真实数据上进行可扩展测试，限制了测试的灵活性和充分性。本文提出 qaAskeR，设计三种新颖的蜕变关系来测试 QA 软件。该方法不依赖标注标签，而是通过递归询问与同一知识相关的多个问题，检查 QA 软件在这些问题上的行为来发现违规。实验结果表明，qaAskeR 无需任何预标注标签即可在超过 80% 的有效案例中揭示违规；在一个最先进的 QA 算法上，它发现了多种回答问题，尤其是跨数据集时对问题类型的泛化能力有限。
Question Answering (QA) is an attractive and challenging area in NLP community. There are diverse algorithms being proposed and various benchmark datasets with different topics and task formats being constructed. QA software has also been widely used in daily human life now. However, current QA software is mainly tested in a reference-based paradigm, in which the expected outputs (labels) of test cases need to be annotated with much human effort before testing. As a result, neither the just-in-time test during usage nor the extensible test on massive unlabeled real-life data is feasible, which keeps the current testing of QA software from being flexible and sufficient. In this paper, we propose a method, qaAskeR, with three novel Metamorphic Relations for testing QA software. qaAskeR does not require the annotated labels but tests QA software by checking its behaviors on multiple recursively asked questions that are related to the same knowledge. Experimental results show that qaAskeR can reveal violations at over 80% of valid cases without using any preannotated labels. Diverse answering issues, especially the limited generalization on question types across datasets, are revealed on a state-of-the-art QA algorithm.
DOI 原文 ·
@inproceedings{paperbot6511,
title = {Testing Your Question Answering Software via Asking Recursively},
author = {Songqiang Chen and Shuo Jin and Xiaoyuan Xie},
booktitle = {ASE},
year = {2021},
doi = {10.1109/ase51524.2021.9678670}
}