一句话：提出面向 AI 的编程语言文法概念，实现保持 AST 等价的 Python 方言 SimPy，减少 LLM 代码生成 token 用量 10% 以上。
随着 LLM 成为编程语言的新受众，现有文法与排版主要为人类可读性设计，含大量语法与格式 token，给模型推理带来额外计算开销。论文提出面向 AI 的文法概念，主张用最少 token 表达代码语义。作者以启发式规则改写 Python 文法，实现首个面向 AI 的 Python 方言 Simple Python（SimPy）：它与标准 Python 保持相同 AST，既可用修改后的 AST 解析器执行，也可与 Python 无缝互转，支持人类与 LLM 协作；论文还探索帮助现有 LLM 理解 SimPy 的方法。实验表明，在相同代码任务上，CodeLlama 与 GPT-4 使用 SimPy 后 token 用量分别减少 13.5% 和 10.4%，性能保持甚至提升。作者呼吁社区进一步研究 AI 导向的编程语言文法。
Artificial Intelligence (AI) models have emerged as another important audience for programming languages alongside humans and machines, as we enter the era of large language models (LLMs). LLMs can now perform well in coding competitions and even write programs like developers to solve various tasks, including mathematical problems. However, the grammar and layout of current programs are designed to cater the needs of human developers -- with many grammar tokens and formatting tokens being used to make the code easier for humans to read. While this is helpful, such a design adds unnecessary computational work for LLMs, as each token they either use or produce consumes computational resources. To improve inference efficiency and reduce computational costs, we propose the concept of AI-oriented grammar.This aims to represent code in a way that better suits the working mechanism of AI models. Code written with AI-oriented grammar discards formats and uses a minimum number of tokens to convey code semantics effectively. To demonstrate the feasibility of this concept, we explore and implement the first AI-oriented grammar for Python, named Simple Python (SimPy). SimPy is crafted by revising the original Python grammar through a series of heuristic rules. Programs written in SimPy maintain identical Abstract Syntax Tree (AST) structures to those in standard Python. This allows for not only execution via a modified AST parser, but also seamless transformation between programs written in Python and SimPy, enabling human developers and LLMs to use Python and SimPy, respectively, when they need to collaborate. We also look into methods to help existing LLMs understand and use SimPy effectively. In the experiments, compared with Python, SimPy enables a reduction in token usage by 13.5% and 10.4% for CodeLlama and GPT-4, respectively, when completing the same set of code-related tasks. Additionally, these models can maintain or even improve their performance when using SimPy instead of Python for these tasks. With these promising results, we call for further contributions to the development of AI-oriented program grammar within our community.
@inproceedings{paperbot8524,
title = {AI Coders Are among Us: Rethinking Programming Language Grammar towards Efficient Code Generation},
author = {Zhensu Sun and Xiaoning Du and Zhou Yang and Li Li and David Lo},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3680347}
}