一句话：提出 CodeFast：用轻量模型 GenGuard 及时终止推理，抑制 Code LLM 生成冗余 token，加速 34%–452% 且不损代码质量。
代码生成旨在根据自然语言需求自动生成代码片段，但 Code LLM 推理时间过长，严重影响实际使用。作者先对多个 Code LLM 在代码生成任务上进行深入初步研究，发现持续生成冗余 token 这一显著效率问题，既损害开发者生产力又造成大量计算浪费。为此提出 CodeFast，其核心是在检测到不必要的冗余 token 时及时终止推理：首先设计自动数据构造框架获得训练数据，然后训练一个可适用于多种编程语言的统一轻量模型 GenGuard，用于预测当前步是否应终止推理，最后用 GenGuard 增强 Code LLM 以加速代码生成推理。在 5 个代表性 Code LLM 与 4 个广泛使用的代码生成数据集上的大量实验表明：CodeFast 可将不同 Code LLM 的推理速度提升 34% 至 452%，且不损害生成代码质量；同时在不同参数设置下表现稳定，并能泛化到未训练过的数据集。
Code generation aims to automatically generate code snippets that meet given natural language requirements and plays an important role in software development. Although Code LLMs have shown excellent performance in this domain, their long generation time poses a signification limitation in practice use. In this paper, we first conduct an in-depth preliminary study with different Code LLMs on code generation task and identify a significant efficiency issue, i.e., continual generation of excess tokens. It harms the developer productivity and leads to huge computational wastes. To address it, we introduce CodeFast, an inference acceleration approach for Code LLMs on code generation. The key idea of CodeFast is to terminate the inference process in time when unnecessary excess tokens are detected. First, we propose an automatic data construction framework to obtain training data. Then, we train a unified lightweight model GenGuard applicable to multiple programming languages to predict whether to terminate inference at the current step. Finally, we enhance Code LLM with GenGuard to accelerate its inference in code generation task. We conduct extensive experiments with CodeFast on five representative Code LLMs across four widely used code generation datasets. Experimental results show that (1) CodeFast can significantly improve the inference speed of various Code LLMs in code generation, ranging form 34% to 452%, without compromising the quality of generated code. (2) CodeFast is stable across different parameter settings and can generalize to untrained datasets. Our code and data are available at https://github.com/DeepSoftwareAnalytics/CodeFast.
DOI 原文 ·
@inproceedings{paperbot8519,
title = {When to Stop? Towards Efficient Code Generation in LLMs with Excess Token Prevention},
author = {Lianghong Guo and Yanlin Wang and Ensheng Shi and Wanjun Zhong and Hongyu Zhang and Jiachi Chen and Ruikai Zhang and Yuchi Ma and Zibin Zheng},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3680343}
}