一句话：提出用缺陷报告提炼的细粒度特征引导 ChatGPT 生成测试程序，并对多个重构引擎进行差分测试。
重构引擎在现代 IDE 中广泛使用，但其复杂实现容易产生错误重构。现有测试方法难以系统触发重构缺陷。本文提出一种基于 ChatGPT 的测试生成方法：先人工分析重构引擎相关缺陷报告与测试用例，构建包含细粒度触发特征的特性库；再根据预定义提示模板和随机选取的特征自动构造提示，要求 ChatGPT 生成具备指定特征的程序；随后将生成程序输入多个重构引擎进行差分测试。初步评估覆盖 IntelliJ IDEA、Eclipse、VSCode-Java 和 NetBeans 四款主流重构引擎，共发现 115 个此前未知缺陷和 28 处重构行为不一致，其中 78 个缺陷已被原开发者确认。该工作首次以已有缺陷派生特征指导重构引擎测试程序生成，并首次将 LLM 用于该任务。
Software refactoring is widely employed to improve software quality. However, conducting refactorings manually is tedious, time-consuming, and error-prone. Consequently, automated and semi-automated tool support is highly desirable for software refactoring in the industry, and most of the main-stream IDEs provide powerful tool support for refactoring. However, complex refactoring engines are prone to errors, which in turn may result in imperfect and incorrect refactorings. To this end, in this paper, we propose a ChatGPT-based approach to testing refactoring engines. We first manually analyze bug reports and test cases associated with refactoring engines, and construct a feature library containing fine-grained features that may trigger defects in refactoring engines. The approach automatically generates prompts according to both predefined prompt templates and features randomly selected from the feature library, requesting ChatGPT to generate test programs with the requested features. Test programs generated by ChatGPT are then forwarded to multiple refactoring engines for differential testing. To the best of our knowledge, it is the first approach in testing refactoring engines that guides test program generation with features derived from existing bugs. It is also the first approach in this line that exploits LLMs in the generation of test programs. Our initial evaluation of four main-stream refactoring engines suggests that the proposed approach is effective. It identified a total of 115 previously unknown bugs besides 28 inconsistent refactoring behaviors among different engines. Among the 115 bugs, 78 have been manually confirmed by the original developers of the tested engines, i.e., IntelliJ IDEA, Eclipse, VScode-Java, and NetBeans.
DOI 原文 ·
@inproceedings{paperbot8905,
title = {Chatgpt-Based Test Generation for Refactoring Engines Enhanced by Feature Analysis on Examples},
author = {Chunhao Dong and Yanjie Jiang and Yuxia Zhang and Yang Zhang and Hui Liu},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00210}
}