一句话：让 Transformer 感知代码的语法结构，使代码补全的下一 token 预测准确率大幅提升。
代码补全（autocomplete）是现代 IDE 的核心功能，其效果取决于底层模型能否把正确的下一个 token 排在候选列表前列。本文以 Transformer 为基座神经架构，研究如何让这一本质上为序列处理而设计的模型感知代码的语法结构，并给出多种将代码结构信息传递给 Transformer 的方式。作者在标准 Python 数据集以及 Facebook 内部 Python 语料上进行了全面实验，比较了多种替代设计选择，最终在下一 token 预测准确率上超越多个当时最先进的系统，领先幅度为 14% 至 18%。论文承诺开源其代码与数据准备流程。
Code prediction, more specifically autocomplete, has become an essential feature in modern IDEs. Autocomplete is more effective when the desired next token is at (or close to) the top of the list of potential completions offered by the IDE at cursor position. This is where the strength of the underlying machine learning system that produces a ranked order of potential completions comes into play. We advance the state-of-the-art in the accuracy of code prediction (next token prediction) used in autocomplete systems. Our work uses Transformers as the base neural architecture. We show that by making the Transformer architecture aware of the syntactic structure of code, we increase the margin by which a Transformer-based system outperforms previous systems. With this, it outperforms the accuracy of several state-of-the-art next token prediction systems by margins ranging from 14% to 18%. We present in the paper several ways of communicating the code structure to the Transformer, which is fundamentally built for processing sequence data. We provide a comprehensive experimental evaluation of our proposal, along with alternative design choices, on a standard Python dataset, as well as on Facebook internal Python corpus. Our code and data preparation pipeline will be available in open source.