一句话：提出可扩展工具 COMEX，从源代码生成并组合 AST、CFG、DFG 等定制化代码视图，服务 ML4SE 表示学习。
学习有效源代码表示对 ML4SE 至关重要。现有 LLM 常把代码当作文本序列，忽视语法结构；CFG、DFG、AST 等代码视图能补充这些信息，但跨语言生成繁琐。作者提出 COMEX：基于 tree-sitter 的代码视图生成与定制框架。它直接处理不一定可编译的源码，支持 Java 和 C#，通过过程内/过程间分析覆盖方法级与程序级片段，并允许创建、组合多种代码视图供 ML 模型使用。因基于支持 40 多种语言的解析器，COMEX 易扩展至其他语言。论文以开源实现和演示展示功能，未报告定量实验，旨在推动源代码表示学习与 ML4SE 研究。
Learning effective representations of source code is critical for any Machine Learning for Software Engineering (ML4SE) system. Inspired by natural language processing, large language models (LLMs) like Codex and CodeGen treat code as generic sequences of text and are trained on huge corpora of code data, achieving state of the art performance on several software engineering (SE) tasks. However, valid source code, unlike natural language, follows a strict structure and pattern governed by the underlying grammar of the programming language. Current LLMs do not exploit this property of the source code as they treat code like a sequence of tokens and overlook key structural and semantic properties of code that can be extracted from code-views like the Control Flow Graph (CFG), Data Flow Graph (DFG), Abstract Syntax Tree (AST), etc. Unfortunately, the process of generating and integrating code-views for every programming language is cumbersome and time consuming. To overcome this barrier, we propose our tool COMEX - a framework that allows researchers and developers to create and combine multiple code-views which can be used by machine learning (ML) models for various SE tasks. Some salient features of our tool are: (i) it works directly on source code (which need not be compilable), (ii) it currently supports Java and C#, (iii) it can analyze both method-level snippets and program-level snippets by using both intra-procedural and inter-procedural analysis, and (iv) it is easily extendable to other languages as it is built on tree-sitter - a widely used incremental parser that supports over 40 languages. We believe this easy-to-use code-view generation and customization tool will give impetus to research in source code representation learning methods and ML4SE. The source code and demonstration of our tool can be found at https://github.com/IBM/tree-sitter-codeviews and https://youtu.be/GER6U87FVbU, respectively.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{DasMMTSCK23,
title = {COMEX: A Tool for Generating Customized Source Code Representations},
author = {Debeshee Das and Noble Saji Mathews and Alex Mathai and Srikanth Tamilselvam and Kranthi Sedamaki and Sridhar Chimalakonda and Atul Kumar},
booktitle = {ASE},
year = {2023},
doi = {10.1109/ase56229.2023.00010}
}