一句话:提出 TransRepair,用变异与蜕变测试检测机器翻译的一致性缺陷并自动修复。
本文针对机器翻译系统输出不一致的问题,提出全自动的测试与修复方法 TransRepair。该方法将变异测试与蜕变测试结合,在无需人工标准译文(human oracle)的前提下构造应产生一致翻译的输入对,从而检测不一致性缺陷。随后,TransRepair 采用概率参考(probability-reference)或交叉参考(cross-reference)策略,以灰盒或黑盒方式对翻译结果进行后处理以修复不一致。作者在 Google Translate 与 Transformer 两个先进翻译系统上评估:生成一致翻译输入对的精度达 99%;Google Translate 与 Transformer 分别约有 36% 和 40% 的不一致缺陷。黑盒修复平均修复 Google Translate 28%、Transformer 19% 的缺陷,灰盒修复平均修复 Transformer 30% 的缺陷。人工检查表明,修复后的翻译在 87% 的情况下提升了一致性(2% 变差),在 27% 的情况下可接受性更好(8% 更差)。
This paper presents TransRepair, a fully automatic approach for testing and repairing the consistency of machine translation systems. TransRepair combines mutation with metamorphic testing to detect inconsistency bugs (without access to human oracles). It then adopts probability-reference or cross-reference to post-process the translations, in a grey-box or black-box manner, to repair the inconsistencies. Our evaluation on two state-of-the-art translators, Google Translate and Transformer, indicates that TransRepair has a high precision (99%) on generating input pairs with consistent translations. With these tests, using automatic consistency metrics and manual assessment, we find that Google Translate and Transformer have approximately 36% and 40% inconsistency bugs. Black-box repair fixes 28% and 19% bugs on average for Google Translate and Transformer. Grey-box repair fixes 30% bugs on average for Transformer. Manual inspection indicates that the translations repaired by our approach improve consistency in 87% of cases (degrading it in 2%), and that our repairs have better translation acceptability in 27% of the cases (worse in 8%).