一句话：提出 EarlyBIRD，用预训练 Transformer 早期层构建代码复合表示，在不增加资源下提升缺陷检测效率与精度。
深度 NLP 模型在漏洞检测、类型推断等软件工程任务上效果显著，但训练和微调开销大。本文提出通用方法 EarlyBIRD，从预训练 Transformer（以 CodeBERT 为例）的早期层提取表示，并组合成代码的复合表示，从而更充分地利用已有资源与信息。作者系统比较了 12 种复合表示构建策略与仅使用最后一层编码器的标准做法，在四个数据集上评估缺陷检测与多分类性能。结果显示，多种早期层组合在缺陷检测上优于基线，部分组合也提升多分类表现：仅用 CodeBERT 12 层中的 3 层，Devign 上的检测准确率平均提升 2 个百分点，微调速度提升 3.3 倍。这表明早期层可在同等资源下取得更好结果，也可显著降低微调与推理开销。
The use of modern Natural Language Processing (NLP) techniques has shown to be beneficial for software engineering tasks, such as vulnerability detection and type inference. However, training deep NLP models requires significant computational resources. This paper explores techniques that aim at achieving the best usage of resources and available information in these models. We propose a generic approach, EarlyBIRD, to build composite representations of code from the early layers of a pre-trained transformer model. We empirically investigate the viability of this approach on the CodeBERT model by comparing the performance of 12 strategies for creating composite representations with the standard practice of only using the last encoder layer. Our evaluation on four datasets shows that several early layer combinations yield better performance on defect detection, and some combinations improve multi-class classification. More specifically, we obtain a +2 average improvement of detection accuracy on Devign with only 3 out of 12 layers of CodeBERT and a 3.3x speed-up of fine-tuning. These findings show that early layers can be used to obtain better results using the same resources, as well as to reduce resource usage during fine-tuning and inference.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2305-04940,
title = {The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classification},
author = {Anastasiia Grishina and Max Hort and Leon Moonen},
booktitle = {FSE},
year = {2023},
doi = {10.1145/3611643.3616304}
}