一句话：通过四类探针任务系统评估预训练代码模型对代码表层、语法、结构与语义信息的编码能力。
预训练代码模型（如 CodeBERT、GraphCodeBERT）在代码生成、摘要等软件工程任务上表现优异，但其向量表示是否充分编码源代码特征、能否广泛迁移到下游任务仍不清楚。本文引入诊断性探针任务，构造了探测表层、语法、结构和语义四类代码信息的探针，用于识别模型在理解哪些代码属性上存在缺陷、刻画不同模型层的表征特性，并考察模型的样本效率。作者对四个预期代码知识水平不同的模型进行探测：仅在英语上预训练的 BERT、在源代码与自然语言文档上预训练的 CodeBERT 和 CodeBERTa、以及在源代码与数据流上预训练的 GraphCodeBERT。结果显示，GraphCodeBERT 整体表现最为稳定一致，但 BERT 在某些代码任务上出乎意料地好，这一现象值得进一步研究。
Pre-trained models of code built on the transformer architecture have performed well on software engineering (SE) tasks such as predictive code generation, code summarization, among others. However, whether the vector representations from these pre-trained models comprehensively encode characteristics of source code well enough to be applicable to a broad spectrum of downstream tasks remains an open question.One way to investigate this is with diagnostic tasks called probes. In this paper, we construct four probing tasks (probing for surface-level, syntactic, structural, and semantic information) for pre-trained code models. We show how probes can be used to identify whether models are deficient in (understanding) certain code properties, characterize different model layers, and get insight into the model sample-efficiency.We probe four models that vary in their expected knowledge of code properties: BERT (pre-trained on English), CodeBERT and CodeBERTa (pre-trained on source code, and natural language documentation), and GraphCodeBERT (pre-trained on source code with dataflow). While GraphCodeBERT performs more consistently overall, we find that BERT performs surprisingly well on some code tasks, which calls for further investigation.