一句话：实证发现 Web GUI 测试脆弱性根因多元，LLM 借助类人诊断上下文可修复多数脆弱测试。
自动化 Web GUI 测试对软件质量至关重要，但在持续演化的真实应用中常因测试脆弱性而失效。本文是一篇经验论文，从实用角度系统探究脆弱性的根因：首先分析 Mind2Web 数据集中遗留测试用例在当前版本 Web 应用上失败的原因，发现脆弱性源于测试脚本设计、Web 应用复杂度与自动化框架局限等多方面因素。纵向研究进一步显示，81.7% 被修复的测试在六个月内再次失效，且多由相似的反复出现的问题导致，说明脆弱性具有持续性。作者还证明，当向大语言模型提供类人诊断上下文时，可以成功修复相当一部分脆弱测试，但在更复杂场景中人类专家仍然不可或缺。研究强调脆弱性是多因素问题，需要自动化测试各参与方协同解决。
Automated web GUI testing is important for software quality, however, its effectiveness is often undermined by test case brittleness, especially in continuously evolving real-world applications. In this experience paper, we pragmatically investigate the root causes of brittleness. We first analyze why legacy test cases, derived from the Mind2Web dataset, fail when executed on current web application versions. Our findings reveal that brittleness stems from multifaceted factors, including test script design, web application complexity, and automation framework limitations. A longitudinal study further shows that 81.7% of repaired tests break again within six months, primarily due to similar recurring issues, highlighting the persistent nature of brittleness. We further demonstrate that Large Language Models, when provided with human-like diagnostic context, can successfully repair a substantial portion of these brittle tests, though human expertise remains important for more complex scenarios. Our findings emphasize that brittleness is a multifaceted problem requiring collaboration between different parts involved in the automation testing.
DOI 原文 ·
@inproceedings{paperbot9253,
title = {Who’s to Blame? Rethinking the Brittleness of Automated Web GUI Testing from a Pragmatic Perspective},
author = {Haonan Zhang and Kundi Yao and Zishuo Ding and Lizhi Liao and Weiyi Shang},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00251}
}