paperbot · PL 论文追踪

全部论文 待读 收藏

P2 Automatic testing and improvement of machine translation

机器翻译的自动测试与改进
ICSE Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering2020匹配度 3/10 · 该论文研究机器翻译系统的一致性测试与自动修复,属于机器学习系统测试/调试方向,但与用户关注的静态分析、Agentic程序、浮点程序分析、贝叶斯程序分析及LLM与PL交叉相距较远;虽涉及变异与蜕变测试,但核心不是程序语言静态分析。引用 112 inbox
Zeyu Sun, Jie M. Zhang, Mark Harman, Mike Papadakis, Lu Zhang
crossref: 2026-09-12

AI 速览

一句话:提出 TransRepair,用变异与蜕变测试检测机器翻译的一致性缺陷并自动修复。

本文针对机器翻译系统输出不一致的问题,提出全自动的测试与修复方法 TransRepair。该方法将变异测试与蜕变测试结合,在无需人工标准译文(human oracle)的前提下构造应产生一致翻译的输入对,从而检测不一致性缺陷。随后,TransRepair 采用概率参考(probability-reference)或交叉参考(cross-reference)策略,以灰盒或黑盒方式对翻译结果进行后处理以修复不一致。作者在 Google Translate 与 Transformer 两个先进翻译系统上评估:生成一致翻译输入对的精度达 99%;Google Translate 与 Transformer 分别约有 36% 和 40% 的不一致缺陷。黑盒修复平均修复 Google Translate 28%、Transformer 19% 的缺陷,灰盒修复平均修复 Transformer 30% 的缺陷。人工检查表明,修复后的翻译在 87% 的情况下提升了一致性(2% 变差),在 27% 的情况下可接受性更好(8% 更差)。

问题
机器翻译系统在面对语义等价但形式不同的输入时可能产生不一致的翻译,而检测此类缺陷通常依赖人工标准译文,成本高且难以规模化。
方法
将变异测试与蜕变测试结合,自动生成应产生一致翻译的输入对以检测不一致缺陷;再用概率参考或交叉参考,以灰盒或黑盒后处理方式修复翻译的不一致。
评估
在 Google Translate 和 Transformer 上评估,用自动一致性指标与人工评估衡量缺陷检测与修复效果,并报告修复后的一致性提升与翻译可接受性变化。
原文摘要(Abstract)

This paper presents TransRepair, a fully automatic approach for testing and repairing the consistency of machine translation systems. TransRepair combines mutation with metamorphic testing to detect inconsistency bugs (without access to human oracles). It then adopts probability-reference or cross-reference to post-process the translations, in a grey-box or black-box manner, to repair the inconsistencies. Our evaluation on two state-of-the-art translators, Google Translate and Transformer, indicates that TransRepair has a high precision (99%) on generating input pairs with consistent translations. With these tests, using automatic consistency metrics and manual assessment, we find that Google Translate and Transformer have approximately 36% and 40% inconsistency bugs. Black-box repair fixes 28% and 19% bugs on average for Google Translate and Transformer. Grey-box repair fixes 30% bugs on average for Transformer. Manual inspection indicates that the translations repaired by our approach improve consistency in 87% of cases (degrading it in 2%), and that our repairs have better translation acceptability in 27% of the cases (worse in 8%).

链接与引用

DOI 原文 · PDF(开放获取) ·

BibTeX
@inproceedings{paperbot5626,
  title = {Automatic testing and improvement of machine translation},
  author = {Zeyu Sun and Jie M. Zhang and Mark Harman and Mike Papadakis and Lu Zhang},
  booktitle = {ICSE},
  year = {2020},
  doi = {10.1145/3377811.3380420}
}