一句话：提出 PatInv，通过生成语义不同但句法相似的句子并检测翻译是否相同，发现机器翻译错误。
神经机器翻译已广泛使用，但常返回错误译文，可能造成经济损失或政治冲突；由于神经模型复杂，测试机器翻译系统面临新挑战。本文提出 PatInv，其核心直觉是含义不同的句子不应得到相同翻译。PatInv 有两种实现：利用掩码语言模型替换句中的一个词，或依据成分结构删除词/短语，从而生成句法相似但语义不同的句子，再检查原句与变体句的翻译是否相同。作者在 Google Translate 和 Bing Microsoft Translator 上用 200 个英语句子测试 En-Hi 与 En-Zh 两种语言设置，分别发现 308 和 223 个错误翻译，其中多数无法被现有最优方法发现。
Machine translation software has become heavily integrated into our daily lives due to the recent improvement in the performance of deep neural networks. However, machine translation software has been shown to regularly return erroneous translations, which can lead to harmful consequences such as economic loss and political conflicts. Additionally, due to the complexity of the underlying neural models, testing machine translation systems presents new challenges. To address this problem, we introduce a novel methodology called PatInv. The main intuition behind PatInv is that sentences with different meanings should not have the same translation. Under this general idea, we provide two realizations of PatInv that given an arbitrary sentence, generate syntactically similar but semantically different sentences by: (1) replacing one word in the sentence using a masked language model or (2) removing one word or phrase from the sentence based on its constituency structure. We then test whether the returned translations are the same for the original and modified sentences. We have applied PatInv to test Google Translate and Bing Microsoft Translator using 200 English sentences. Two language settings are considered: English-Hindi (En-Hi) and English-Chinese (En-Zh). The results show that PatInv can accurately find 308 erroneous translations in Google Translate and 223 erroneous translations in Bing Microsoft Translator, most of which cannot be found by the state-of-the-art approaches.
DOI 原文 ·
@inproceedings{paperbot5740,
title = {Machine translation testing via pathological invariance},
author = {Shashij Gupta and Pinjia He and Clara Meister and Zhendong Su},
booktitle = {FSE},
year = {2020},
doi = {10.1145/3368089.3409756}
}