一句话：提出按自主程度划分的 LLM 测试智能体分类法，并展示对话式框架如何辅助软件测试。
软件测试是开发周期中的关键环节，但需要专门知识并耗费开发者大量精力。近期大语言模型（LLM）展现出的能力提示其可作为自动化测试助手，甚至驱动测试流程。本文提出一个基于自主程度（level of autonomy）的 LLM 测试智能体分类法，论述更高自主性在实践中的价值，并给出一个将 LLM 用作测试助手的示例，说明对话式测试框架如何帮助开发者。文中特别指出，LLM 常被诟病的“幻觉”在测试场景中反而可能带来益处，因为其能生成开发者未曾设想的测试输入或场景。作者还梳理了 LLM 驱动的测试智能体可带来的其他实际收益，并讨论了其潜在局限。
Software testing is an important part of the development cycle, yet it requires specialized expertise and substantial developer effort to adequately test software. Recent discoveries of the capabilities of large language models (LLMs) suggest that they can be used as automated testing assistants, and thus provide helpful information and even drive the testing process. To highlight the potential of this technology, we present a taxonomy of LLM-based testing agents based on their level of autonomy, and describe how a greater level of autonomy can benefit developers in practice. An example use of LLMs as a testing assistant is provided to demonstrate how a conversational framework for testing can help developers. This also highlights how the often criticized “hallucination” of LLMs can be beneficial for testing. We identify other tangible benefits that LLM-driven testing agents can bestow, and also discuss potential limitations.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{FeldtKYY23,
title = {Towards Autonomous Testing Agents via Conversational Large Language Models},
author = {Robert Feldt and Sungmin Kang and Juyeon Yoon and Shin Yoo},
booktitle = {ASE},
year = {2023},
doi = {10.1109/ase56229.2023.00148}
}