一句话：首次研究并测试 NMT 系统效率鲁棒性，通过极小扰动延迟 EOS 显著增加延迟与能耗。
论文关注神经机器翻译（NMT）系统在高吞吐与实时需求下的计算效率鲁棒性，而现有工作多聚焦准确率及其鲁棒性。作者分析 1455 个公开 NMT 系统，发现输出长度而非输入决定计算效率，输出长度受预配置最大迭代阈值和运行时生成的句末（EOS）token 影响。为此提出 NMTSloth，用梯度引导搜索在字符级、token 级和结构级寻找最小且不易察觉的扰动，延迟 EOS 出现，迫使输入达到原本不可达的阈值。在 Google T5、AllenAI WMT14 与 Helsinki-NLP 上评估，仅改动一个字符或 token 即将响应延迟提高 85%–3153%、能耗提高 86%–3052%；案例研究显示真实移动设备电池耗电可达正常输入的 30 倍以上。
Neural Machine Translation (NMT) systems have received much recent attention due to their human-level accuracy. While existing works mostly focus on either improving accuracy or testing accuracy robustness, the computation efficiency of NMT systems, which is of paramount importance due to often vast translation demands and real-time requirements, has surprisingly received little attention. In this paper, we make the first attempt to understand and test potential computation efficiency robustness in state-of-the-art NMT systems. By analyzing the working mechanism and implementation of 1455 public-accessible NMT systems, we observe a fundamental property in NMT systems that could be manipulated in an adversarial manner to reduce computation efficiency significantly. Our interesting observation is that the output length determines the computation efficiency of NMT systems instead of the input, where the output length depends on two factors: an often sufficiently large yet pessimistic pre-configured threshold controlling the max number of iterations and a runtime generated end of sentence (EOS) token. Our key motivation is to generate test inputs that could sufficiently delay the generation of EOS such that NMT systems would have to go through enough iterations to satisfy the pre-configured threshold. We present NMTSloth, which develops a gradient-guided technique that searches for a minimal and unnoticeable perturbation at character-level, token-level, and structure-level, which sufficiently delays the appearance of EOS and forces these inputs to reach the naturally-unreachable threshold. To demonstrate the effectiveness of NMTSloth, we conduct a systematic evaluation on three public-available NMT systems: Google T5, AllenAI WMT14, and Helsinki-NLP translators. Experimental results show that NMTSloth can increase NMT systems' response latency and energy consumption by 85% to 3153% and 86% to 3052%, respectively, by perturbing just one character or token in the input sentence. Our case study shows that inputs generated by NMTSloth significantly affect the battery power in real-world mobile devices (i.e., drain more than 30 times battery power than normal inputs).
@inproceedings{paperbot7088,
title = {NMTSloth: understanding and testing efficiency degradation of neural machine translation systems},
author = {Simin Chen and Cong Liu and Mirazul Haque and Zihe Song and Wei Yang},
booktitle = {FSE},
year = {2022},
doi = {10.1145/3540250.3549102}
}