一句话：提出 Token Sugar：将高频冗长代码模式替换为可逆简写，降低 LLM 代码任务的 token 开销并保持性能。
大语言模型在代码生成与理解上表现优异，但编程语言本身的冗长（格式化元素、样板代码）导致输入与输出的 token 数量膨胀，推高推理成本并拖慢生成。已有工作通过简化编程语言语法来减少 token，但局限于句法层面，语义层面的压缩空间未被挖掘。本文提出 Token Sugar 概念：把高频且冗长的代码模式替换为可逆、token 高效的简写形式。作者设计了系统化方案，从代码语料中挖掘高频、token 重的模式，为每个模式映射唯一简写，并通过代码变换将其融入 LLM 预训练。该方案得到 799 个（代码模式, 简写）对，可在源代码上减少最多 15.1% 的 token 数，且与既有的以语法为中心的方法互补。作者用 Token Sugar 增强的数据训练三个常用 LLM，实验显示生成阶段 token 最多减少 11.2%，同时 Pass@1 与在未处理代码上训练的基线几乎持平。
Large language models (LLMs) have shown exceptional performance in code generation and understanding tasks, yet their high computational costs hinder broader adoption. One important factor is the inherent verbosity of programming languages, such as unnecessary formatting elements and lengthy boilerplate code. This leads to inflated token counts in both input and generated outputs, which increases inference costs and slows down the generation process. Prior work improves this through simplifying programming language grammar, reducing token usage across both code understanding and generation tasks. However, it is confined to syntactic transformations, leaving significant opportunities for token reduction unrealized at the semantic level.In this work, we propose Token Sugar, a concept that replaces frequent and verbose code patterns with reversible, token-efficient shorthand in the source code. To realize this concept in practice, we designed a systematic solution that mines high-frequency, token-heavy patterns from a code corpus, maps each to a unique shorthand, and integrates them into LLM pretraining via code transformation. With this solution, we obtain 799 (code pattern, shorthand) pairs, which can reduce up to 15.1% token count in the source code and is complementary to existing syntax-focused methods. We further trained three widely used LLMs on Token Sugar-augmented data. Experimental results show that these models not only achieve significant token savings (up to 11.2% reduction) during generation but also maintain near-identical Pass@1 scores compared to baselines trained on unprocessed code.
DOI 原文 ·
@inproceedings{paperbot9398,
title = {Token Sugar: Making Source Code Sweeter for LLMs through Token-Efficient Shorthand},
author = {Zhensu Sun and Chengran Yang and Xiaoning Du and Zhou Yang and Li Li and David Lo},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00201}
}