一句话：提出 SelfPiCo，用 Code Llama 在交互循环中自引导执行部分代码，显著提升可执行率并检测运行时错误。
部分或任意代码执行对软件调试与测试至关重要，但常因缺少定义和复杂第三方依赖而难以执行。现有方法用机器学习预测未定义元素类型并注入预定义 dummy 值，但受限于简单 dummy 值和无法持续学习。本文设计并实现 SelfPiCo，将开源 LLM（Code Llama）嵌入交互循环来动态引导部分代码执行。它利用 few-shot in-context learning 和 chain-of-thought 推理，并基于微调 Code Llama 持续从执行结果中学习、逐步修正预测。评估表明，SelfPiCo 在开源代码和 Stack Overflow 片段中分别可执行 72.7% 和 83.3% 的代码行，比最新 Lexecutor 高出 37.9% 和 33.5%；并在八个 GitHub 软件项目和 43 个 Stack Overflow 帖子中分别检测到 18 和 33 个运行时类型错误问题。
Code executability plays a vital role in software debugging and testing (e.g., detecting runtime exceptions or assertion violations). However, code execution, especially partial or arbitrary code execution, is a non-trivial task due to missing definitions and complex third-party dependencies. To make partial code (such as code snippets posted on the web or code fragments deep inside complex software projects) executable, the existing study has proposed a machine learning model to predict the undefined element types and inject the pre-defined dummy values into execution. However, the performance of their tool is limited due to its simply designed dummy values and the inability to continue learning. In this paper, we design and implement a novel framework, named SelfPiCo (Self-Guided Partial Code Executor), to dynamically guide partial code execution by incorporating the open-source LLM (i.e., Code Llama) within an interactive loop. Particularly, SelfPiCo leverages few-shot in-context learning and chain-of-thought reasoning to elicit human knowledge and logical reasoning based on fine-tuning the Code Llama model. SelfPiCo continuously learns from code execution results and refines its predictions step after step. Our evaluations demonstrate that SelfPiCo can execute 72.7% and 83.3% of all lines in the open-source code and Stack Overflow snippets, outperforming the most recent state-of-the-art Lexecutor by 37.9% and 33.5%, respectively. Moreover, SelfPiCo successfully detected 18 and 33 runtime type error issues by executing the partial code from eight GitHub software projects and 43 Stack Overflow posts, demonstrating the practical usage and potential application of our framework in practice.
DOI 原文 ·
@inproceedings{paperbot8459,
title = {SelfPiCo: Self-Guided Partial Code Execution with LLMs},
author = {Zhipeng Xue and Zhipeng Gao and Shaohua Wang and Xing Hu and Xin Xia and Shanping Li},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3680368}
}