一句话：提出 DietCode，通过简化输入程序使 CodeBERT 在计算成本降低 40% 的同时保持性能相当。
预训练代码表示模型（如 CodeBERT）在多种软件工程任务中表现优异，但其计算复杂度随输入序列长度呈二次增长，难以轻量化部署。本文通过分析 CodeBERT 的注意力分布，发现模型更关注关键词和数据相关语句等特定词元与语句。基于此，作者提出 DietCode，采用词丢弃、频率过滤以及基于注意力的语句/词元选择三种策略来简化输入程序，从而在不损害模型性能的前提下大幅降低计算开销。在两个下游任务上的实验表明，DietCode 在微调和测试阶段减少 40% 计算成本的情况下，取得了与原始 CodeBERT 相当的性能。
Pre-trained code representation models such as CodeBERT have demonstrated superior performance in a variety of software engineering tasks, yet they are often heavy in complexity, quadratically with the length of the input sequence. Our empirical analysis of CodeBERT's attention reveals that CodeBERT pays more attention to certain types of tokens and statements such as keywords and data-relevant statements. Based on these findings, we propose DietCode, which aims at lightweight leverage of large pre-trained models for source code. DietCode simplifies the input program of CodeBERT with three strategies, namely, word dropout, frequency filtering, and an attention-based strategy that selects statements and tokens that receive the most attention weights during pre-training. Hence, it gives a substantial reduction in the computational cost without hampering the model performance. Experimental results on two downstream tasks show that DietCode provides comparable results to CodeBERT with 40% less computational cost in fine-tuning and testing.