一句话：提出以蜕变测试与蜕变关系检验词性标注工具的缺陷，并在 spaCy、NLTK、Flair 上实证有效。
词性标注是自然语言处理的基础技术，其预测结果为情感分析、词义消歧、信息检索等下游任务提供支撑，但标注工具的错误预测会向后传播，在关键领域甚至造成严重威胁。本文提出用蜕变测试（Metamorphic Testing）检验词性标注工具应满足的性质，以此揭示其缺陷。作者初步设计了两组蜕变关系，分别对应状语从句与句子追加这两类句内与句间变换，并在三种常见工具 spaCy、NLTK 和 Flair 上进行探索性实验。结果表明，即便面对相当简单的变换，该方法也能有效暴露这些工具的处理缺陷，显示出系统化测试并发现未知问题的潜力，可服务于词性标注工具的验证、修复与改进。
Part-of-Speech (POS) tagging for sentences is a basic and widely-used Natural Language Processing (NLP) technique. People rely heavily on it to predict POS tags that serve as the base for many advanced NLP tasks, such as sentiment analysis, word sense disambiguation, and information retrieval. However, POS tagging tools could make wrong predictions, which bring consequent error propagation to the advanced tasks and even cause serious threats in critical application domains. In this paper, we propose to test POS tagging tools with Metamorphic Testing against some properties that they should follow. The preliminary exploration with two groups of Metamorphic Relations shows that our method can effectively reveal defects of three common POS tagging tools (i.e., spaCy, NLTK, and Flair) on handling fairly simple intra- and inter-sentence transformation regarding adverbial clause and sentence appending. This demonstrates the great potential of our method to deliver a systematic test and reveal the unaware issues, which may benefit the validation, repair, and improvement, for POS tagging tools.
DOI 原文 ·
@inproceedings{paperbot6566,
title = {Property-based Test for Part-of-Speech Tagging Tool},
author = {Shuo Jin and Songqiang Chen and Xiaoyuan Xie},
booktitle = {ASE},
year = {2021},
doi = {10.1109/ase51524.2021.9678807}
}