一句话：首个用 LLM Agent 修复非幂等结果 flaky 测试的框架，经检测、探索、修复三阶段定位状态污染并生成补丁。
Flaky 测试在重复执行时结果不一致，给回归测试带来严重困扰。其中非幂等结果（NIO）flaky 测试指首次运行通过、但在同一环境下后续运行失败的测试。现有基于大语言模型（LLM）的 flaky 测试修复方法尚未处理 NIO 情况，且受限于 LLM 的上下文窗口，难以纳入测试方法之外的相关源码，从而忽略状态污染这一 NIO 根源信息。本文提出 NIODebugger，首个利用 LLM Agent 修复 flaky 测试的框架，包含检测、探索与修复三个阶段：检测阶段通过动态分析多次运行的栈轨迹与自定义执行日志刻画累积状态污染；探索阶段由 LLM Agent 指导抽取与 flakiness 相关的源码；修复阶段综合利用前述信息生成测试补丁。NIODebugger 可接入多种 LLM，补丁成功率为 11.63% 至 58.72%；最佳变体 NIODebugger-GPT-4 在 20 个大型开源项目中为 172 个此前未知的 NIO 测试中的 101 个生成正确补丁，提交的 PR 中 58 个被合并、1 个被拒、42 个待处理。其 Java 实现以 Maven 插件形式发布。
Flaky tests, characterized by inconsistent results across repeated executions, present significant challenges in software testing, especially during regression testing. Recently, there has been emerging research interest in non-idempotentoutcome (NIO) flaky tests-tests that pass on the initial run but fail on subsequent executions within the same environment. Despite progress in utilizing Large Language Models (LLMs) to address flaky tests, existing methods have not tackled NIO flaky tests. The limited context window of LLMs restricts their ability to incorporate relevant source code beyond the test method itself, often overlooking crucial information needed to address state pollution, which is the root cause of NIO flakiness. This paper introduces NIODebugger, the first framework to utilize an LLM-based agent to repair flaky tests. NIODebugger features a three-phase design: detection, exploration, and fixing. In the detection phase, dynamic analysis collects stack traces and custom test execution logs from multiple test runs, which helps in understanding accumulative state pollution. During the exploration phase, the LLM-based agent provides instructions for extracting relevant source code associated with test flakiness. In the fixing phase, NIODebugger repairs the tests using the information gathered from the previous phases. NIODebugger can be integrated with multiple LLMs, achieving patching success rates ranging from 11.63% to 58.72%. Its best-performing variant, NIODebugger-GPT-4, successfully generated correct patches for 101 out of 172 previously unknown NIO tests across 20 largescale open-source projects. We submitted pull requests for all generated patches; 58 have been merged, only 1 was rejected, and the remaining 42 are pending. The Java implementation of NIODebugger is provided as a Maven plugin accessible at https://github.com/kaiyaok2/NIOInspector.
DOI 原文 ·
@inproceedings{paperbot8834,
title = {NIODebugger: A Novel Approach to Repair Non-Idempotent-Outcome Tests with LLM-Based Agent},
author = {Kaiyao Ke},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00226}
}