一句话：通过挖掘开源项目与访谈从业者，实证揭示LLM智能体应用测试以单元测试为主、覆盖不足并构建23种测试模式分类。
本文实证研究基于LLM的智能体应用测试实践。作者挖掘开源项目，构建应用、工具与测试数据集，人工标注来自240个模块的2572个测试方法，归纳出23种测试模式并覆盖fixture、数据、目标与断言，按单元、模块、集成层级刻画测试；同时访谈10名资深工业实践者。结果显示，测试以范围狭窄的单元测试为主，复杂交互、真实场景与非功能需求覆盖不足，依赖简单输入、重度mock和浅层验证，结构复杂度低；工业界更重视非功能测试，但两者均有形式化基础缺失、目标不清、高质量测试数据难生成等缺口。论文据此提出智能体可测试性、形式化测试目标与故障驱动测试等方向，并发布精选数据集。
Agents built on large language models (LLMs) are increasingly used to develop applications that perform complex, multi-step tasks involving reasoning, tool use, and interaction with external environments. Despite rapid advances in benchmarking LLM-based agents, little is known about how they are tested in practice. In particular, key aspects of testing---such as testing levels, objectives, data patterns, test complexity, and validation strategies---for agent applications remain underexplored. In this paper, we present an empirical study of testing practices for LLM-based agent applications using a corpus of mined open-source projects. We construct an extensive dataset of agent applications, tools, and tests, and manually label 2,572 test methods from 240 modules. From this analysis, we develop a taxonomy of 23 testing patterns across test fixtures, data, objectives, and assertions, and characterize tests by level (unit, module, integration). We complement this with structured interviews of 10 senior industry practitioners building agentic systems. Our results show that testing of LLM-based agent applications is dominated by narrowly scoped unit tests, with limited coverage of complex interactions, realistic scenarios, and non-functional requirements. Tests often rely on simplistic inputs, heavy mocking, and shallow validation, and exhibit low structural complexity. Although industry practice places greater emphasis on non-functional testing than open-source projects, both reveal common gaps, including the lack of formal testing foundations, unclear test objectives, and challenges in generating high-quality test data. Based on these findings, we identify research directions toward more systematic and rigorous testing of agent applications, including foundations for agent testability, formalized test objectives, and fault-based testing techniques tailored to agentic systems. We also release a curated dataset of testing artifacts to support future research.
@inproceedings{paperbot12530,
title = {Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications},
author = {Rangeet Pan and Tyler Stennett and Divya Sankar and Bridget McGinn and Alessandro Orso and Raju Pavuluri and Saurabh Sinha and Maja Vukovic},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3837414}
}