一句话：首次提出将RNN分解为模块的方法，让训练好的RNN可复用、可替换且无需重训。
本文研究如何将训练好的循环神经网络（RNN）分解为模块，以便无需从头重训即可扩展新语言或替换故障部分。已有面向全连接网络和卷积网络的分解方法，但RNN的层结构、循环结构、输入输出架构及激活函数不同，无法直接适用。作者提出首个RNN模块分解方法，覆盖Vanilla、LSTM和GRU，并展示模块的复用与替换。评估使用Math QA、Brown Corpus、Wiki-toxicity、Cline OOS和Tatoeba五个数据集，每个数据集四个模型变体。结果显示分解代价很小（准确率下降0.6%，BLEU提升0.10%），分解出的模块无需重训即可复用和替换。
Can we take a recurrent neural network (RNN) trained to translate between languages and augment it to support a new natural language without retraining the model from scratch? Can we fix the faulty behavior of the RNN by replacing portions associated with the faulty behavior? Recent works on decomposing a fully connected neural network (FCNN) and convolutional neural network (CNN) into modules have shown the value of engineering deep models in this manner, which is standard in traditional SE but foreign for deep learning models. However, prior works focus on the image-based multi-class classification problems and cannot be applied to RNN due to (a) different layer structures, (b) loop structures, (c) different types of input-output architectures, and (d) usage of both non-linear and logistic activation functions. In this work, we propose the first approach to decompose an RNN into modules. We study different types of RNNs, i.e., Vanilla, LSTM, and GRU. Further, we show how such RNN modules can be reused and replaced in various scenarios. We evaluate our approach against 5 canonical datasets (i.e., Math QA, Brown Corpus, Wiki-toxicity, Cline OOS, and Tatoeba) and 4 model variants for each dataset. We found that decomposing a trained model has a small cost (Accuracy: -0.6%, BLEU score: +0.10%). Also, the decomposed modules can be reused and replaced without needing to retrain.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{ImtiazBSPCR23,
title = {Decomposing a Recurrent Neural Network into Modules for Enabling Reusability and Replacement},
author = {Sayem Mohammad Imtiaz and Fraol Batole and Astha Singh and Rangeet Pan and Breno Dantas Cruz and Hridesh Rajan},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00093}
}