一句话:研究LLM生成单元测试的修复循环,构建失败到修复的测试对数据集,归纳修复引入问题的分类法并比较提示策略。
大型语言模型(LLM)流水线能生成语义合理且覆盖率高的单元测试,但生成的测试常无法编译或执行。为此,许多框架采用LLM驱动的修复循环,在失败时迭代重新提示直至验证通过,然而这类修复可能引入新问题,例如为让测试通过而虚构一个不存在的类。本文针对三个Java项目研究修复引入的问题及提示策略的影响。作者构建了一个LLM生成测试的“坏到好”修复对数据集,其中初始失败的测试在zero-shot、few-shot和chain-of-thought提示下经循环修复。基于对117个修复对的人工分析,论文归纳出包含三类反模式和七种场景的分类法,并通过修复率与修复后覆盖率定量比较三种提示策略的性能。
Large Language Model (LLM) pipelines can generate semantically meaningful unit tests with high coverage, but generated tests often fail to compile or execute. Many frameworks therefore use LLM-driven repair loops that iteratively re-prompt on failures until validation succeeds, but these repairs can introduce issues such as hallucinating an invented class solely to make the test pass. In this paper, we study repair-induced issues and effects of prompt strategy across three Java projects. We build a dataset of LLM-generated broken-to-fixed test pairs, where initially failing tests are repaired via a loop under zero-shot, few-shot, and chain-of-thought prompting, derive a taxonomy from 117 manually analyzed pairs (three anti-pattern categories, seven scenarios), and compare the performance of prompting strategies through repair rate and post-repair coverage.