一句话：通过构建 CodeSem 数据集，实证评估四种深度学习代码模型在别名预测与程序等价性预测两个基础程序分析任务上的表现。
深度神经网络已在众多编程语言任务上取得领先效果，但其在基础程序分析任务中的有效性仍缺乏系统研究。本文开展实证研究，在两个基础任务上评估四种代表性代码模型：CuBERT、CodeBERT、GGNN 与 Graph Sandwiches。任务一为别名预测，判断两个指针是 must alias、may alias 还是 must not alias；任务二为等价性预测，判断两个程序是否语义等价。研究的核心是 CodeSem 数据集，它基于 Linux Kernel、GCC、MySQL 等真实旗舰软件源码构建，并经过人工校验。结果表明所有模型在两个任务上均较准确，CuBERT 尤为突出，别名预测与等价性预测准确率分别达 89% 与 84%。作者进一步深入分析各模型表现，认为深度学习模型总体上能够胜任基础程序分析任务，但在特定情形下其弱点也较为明显。代码与评测数据已公开。
While deep neural networks provide state-of-the-art solutions to a wide range of programming language tasks, their effectiveness in dealing with foundational program analysis tasks remains under explored. In this paper, we present an empirical study that evaluates four prominent models of code (i.e., CuBERT, CodeBERT, GGNN, and Graph Sandwiches) in two such foundational tasks: (1) alias prediction, in which models predict whether two pointers must alias, may alias or must not alias; and (2) equivalence prediction, in which models predict whether or not two programs are semantically equivalent. At the core of this study is CodeSem, a dataset built upon the source code of real-world flagship software (e.g., Linux Kernel, GCC, MySQL) and manually validated for the two prediction tasks. Results show that all models are accurate in both prediction tasks, especially CuBERT with an accuracy of 89% and 84% in alias prediction and equivalence prediction, respectively. We also conduct a comprehensive, in-depth analysis of the results of all models in both tasks, concluding that deep learning models are generally capable of performing foundational tasks in program analysis even though in specific cases their weaknesses are also evident. Our code and evaluation data are publicly available at https://github.com/CodeSemDataset/CodeSem.
@article{paperbot2563,
title = {Evaluating the Effectiveness of Deep Learning Models for Foundational Program Analysis Tasks},
author = {Qian Chen and Chenyang Yu and Ruyan Liu and Chi Zhang and Yu Wang and Ke Wang and Ting Su and Linzhang Wang},
journal = {Proceedings of the ACM on Programming Languages},
volume = {8},
number = {OOPSLA1},
year = {2024},
doi = {10.1145/3649829}
}