一句话：实证分析指令生成与多步推理对代码智能任务的作用，并提出融合最佳方法的自动提示生成方案。
大型代码模型在代码智能任务中效果受提示质量影响，但现有提示多由人工设计，耗时且依赖模型与任务。本文系统研究自动提示生成中的指令生成（IG）与多步推理（MSR），在四种开源LCM和代码翻译、代码摘要、API推荐三个任务上评估常用APG方法。结果表明两部件均显著优于基础提示；作者进而组合最佳IG与MSR提出新APG方法。在开源任务上，代码翻译的CodeBLEU平均提升28.38%，代码摘要ROUGE-L提升58.11%，API推荐SuccessRate@1提升84.53%；在微信WeChat-Bench上API推荐的MRR平均提升148.89%。
Large Code Models (LCMs) have demonstrated potential in advancing various code intelligence tasks. However, their effectiveness can be greatly influenced by the quality of the prompts. Current prompt design strategies in code intelligence studies are mostly manually generated, which could be time-consuming and extremely rely on the base LCMs and tasks. Although automated prompt generation (APG) has been investigated in the natural language processing field, it has not attracted sufficient attention and been well explored in the code intelligence tasks. Considering the various tasks and black-box nature of LCMs faced by developers in practice, it is essential to automate the prompt generation process.To mitigate the gap, we empirically investigate the two important parts in APG, including Instruction Generation (IG) and Muti-Step Reasoning (MSR). The instruction generation part aims at providing a task-related description for instructing LCMs to effectively accomplish specific tasks; while the multistep reasoning part aims at guiding LCMs to produce a series of logical steps before arriving at the final answer. For each part, we evaluate the widely-used APG methods on four open-source LCMs and three code intelligence tasks, i.e., code translation (PL-PL), code summarization (PL-NL) and API recommendation (NL-PL). Experimental results indicate that the two parts in APG can dramatically enhance the performance of the code intelligence tasks compared with the basic prompts. Based on the results, we further propose a novel APG approach by combining the best methods of the two studied parts of APG. Experiments show that the proposed APG approach achieves an average improvement of 28.38% with respect to CodeBLEU for the code translation, 58.11% in terms of ROUGE-L for the code summarization and 84.53% in SuccessRate@1 for the API recommendation over the basic prompts, respectively. To validate the effectiveness in industrial scenario, we further evaluate our approach on WeChat-Bench, a proprietary dataset from the WeChat Group in Tencent for API recommendation, achieving an average improvement of 148.89% in MRR.
DOI 原文 ·
@inproceedings{paperbot9250,
title = {Automated Prompt Generation for Code Intelligence: An Empirical study and Experience in WeChat},
author = {Kexing Ji and Shiyun Fu and Cuiyun Gao and Yujia Chen and Zezhou Yang and Chaozheng Wang and Yuetang Deng},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00285}
}