一句话：提出 POSIT，用 biLSTM-CRF 对混合代码与自然语言文本联合进行语言识别及 AST/词性标注。
开发者交流常混合源代码和自然语言，如 Stack Overflow 与开发者邮件列表；对这类混合文本打标签是可追踪性和精确抽取代码片段以实现复用的关键。本文借鉴自然语言处理中的 code-switching 技术，提出 POSIT：训练带条件随机场（CRF）输出层的 biLSTM，利用 CLANG 编译器产生的抽象语法树（AST）标签和 Stanford 词性标注器的 POS 标签，同时预测混合文本中 token 的源语言、源代码 token 的 AST 标签以及自然语言词的 POS 标签。评估显示，POSIT 将语言识别准确率较当时最优提升 10.6%，将 PoS/AST 标注准确率提升 23.7%。
Software developers use a mix of source code and natural language text to communicate with each other: Stack Overflow and Developer mailing lists abound with this mixed text. Tagging this mixed text is essential for making progress on two seminal software engineering problems --- traceability, and reuse via precise extraction of code snippets from mixed text. In this paper, we borrow code-switching techniques from Natural Language Processing and adapt them to apply to mixed text to solve two problems: language identification and token tagging. Our technique, POSIT, simultaneously provides abstract syntax tree tags for source code tokens, part-of-speech tags for natural language words, and predicts the source language of a token in mixed text. To realize POSIT, we trained a biLSTM network with a Conditional Random Field output layer using abstract syntax tree tags from the CLANG compiler and part-of-speech tags from the Standard Stanford part-of-speech tagger. POSIT improves the state-of-the-art on language identification by 10.6% and PoS/AST tagging by 23.7% in accuracy.
DOI 原文 ·
@inproceedings{paperbot5624,
title = {POSIT},
author = {Profir-Petru Pârtachi and Santanu Kumar Dash and Christoph Treude and Earl T. Barr},
booktitle = {ICSE},
year = {2020},
doi = {10.1145/3377811.3380440}
}