一句话：用 LLM 构建用户任务与故障知识图谱，并以思维链推理生成高可行性探索性测试场景。
探索性测试依赖测试人员的知识与创造力，从终端用户视角设计测试用例以发现系统级缺陷。已有工作基于系统知识图谱（从缺陷报告中抽取场景与判定知识）生成测试场景来支持探索性测试，但缺陷报告质量参差、表达风格多样，且生成的场景往往不可行，导致该方法难以落地。本文利用大语言模型的自然语言理解能力构建“用户任务与故障系统知识图谱”（SysKG-UTF），并借助图谱中的系统与缺陷知识以及 LLM 的逻辑推理能力，生成高可行性与高连贯性的测试场景。作者特别设计了思维链（CoT）推理，从 LLM 中抽取类人知识与逻辑推理，模拟开发者验证测试场景可行性的过程。评估表明该方法显著提升了知识图谱构建质量，尤其对低质量缺陷报告效果明显，生成的测试场景具备高可行性与连贯性；用户研究进一步验证其有效性：8 名参与者两小时内从 8 个种子缺陷中发现 36 个缺陷，显著优于最先进基线方法的 21 个。
Exploratory testing leverages the tester's knowledge and creativity to design test cases for effectively uncovering system-level bugs from the end user's perspective. Researchers have worked on test scenario generation to support exploratory testing based on a system knowledge graph, enriched with scenario and oracle knowledge from bug reports. Nevertheless, the adoption of this approach is hindered by difficulties in handling bug reports of inconsistent quality and varied expression styles, along with the infeasibility of the generated test scenarios. To overcome these limitations, we utilize the superior natural language understanding (NLU) capabilities of Large Language Models (LLMs) to construct a System KG of User Tasks and Failures (SysKG-UTF). Leveraging the system and bug knowledge from the KG, along with the logical reasoning capabilities of LLMs, we generate test scenarios with high feasibility and coherence. Particularly, we design chain-of-thought (CoT) reasoning to extract human-like knowledge and logical reasoning from LLMs, simulating a developer's process of validating test scenario feasibility. Our evaluation shows that our approach significantly enhances the KG construction, particularly for bug reports with low quality. Furthermore, our approach generates test scenarios with high feasibility and coherence. The user study further proves the effectiveness of our generated test scenarios in supporting exploratory testing. Specifically, 8 participants find 36 bugs from 8 seed bugs in two hours using our test scenarios, a significant improvement over the 21 bugs found by the state-of-the-art baseline.
@inproceedings{paperbot11311,
title = {Enhancing Exploratory Testing by Large Language Model and Knowledge Graph},
author = {Yanqi Su and Dianshu Liao and Zhenchang Xing and Qing Huang and Mulong Xie and Qinghua Lu and Xiwei Xu},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3639157}
}