一句话：提出 AST-Probe 探针，从预训练代码语言模型隐藏表示中恢复完整抽象语法树，并证明句法子空间存在且维度较低。
预训练语言模型已广泛用于代码建模，但此前探针研究只揭示简单语言属性，未验证模型是否编码完整编程语言语法结构。本文证明模型隐藏表示中存在包含语法信息的句法子空间，提出 AST-Probe 探针方法，从表示中提取该子空间并恢复输入代码片段的完整抽象语法树。在五个 SOTA 预训练语言模型上的实验表明，该句法子空间普遍存在；中间层编码最多 AST 信息；其最优维度远低于模型表示空间维度，说明模型仅用一小部分表示空间编码编程语言语法信息。
The objective of pre-trained language models is to learn contextual representations of textual data. Pre-trained language models have become mainstream in natural language processing and code modeling. Using probes, a technique to study the linguistic properties of hidden vector spaces, previous works have shown that these pre-trained language models encode simple linguistic properties in their hidden representations. However, none of the previous work assessed whether these models encode the whole grammatical structure of a programming language. In this paper, we prove the existence of a syntactic subspace, lying in the hidden representations of pre-trained language models, which contain the syntactic information of the programming language. We show that this subspace can be extracted from the models’ representations and define a novel probing method, the AST-Probe, that enables recovering the whole abstract syntax tree (AST) of an input code snippet. In our experimentations, we show that this syntactic subspace exists in five state-of-the-art pre-trained language models. In addition, we highlight that the middle layers of the models are the ones that encode most of the AST information. Finally, we estimate the optimal size of this syntactic subspace and show that its dimension is substantially lower than those of the models’ representation spaces. This suggests that pre-trained language models use a small part of their representation spaces to encode syntactic information of the programming languages.
@inproceedings{paperbot7261,
title = {AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language models},
author = {José Antonio Hernández López and Martin Weyssow and Jesús Sánchez Cuadrado and Houari Sahraoui},
booktitle = {ASE},
year = {2022},
doi = {10.1145/3551349.3556900}
}