一句话：提出 Reca11，用 LLM 从用户测试录像的自动转录中提取可访问性与可用性问题，自动生成测试报告。
可访问性测试是软件开发的重要环节，自动化工具虽能检测部分问题，但由不同残障测试者参与的用户测试最为全面有效。然而开发者对用户测试望而却步：需要开展冗长的访谈，并事后从大量录像中梳理出待修复的问题。本文探索利用 GPT-4.0 等大语言模型（LLM）在上下文理解与语义文本生成上的能力来缓解这一痛点。作者提出 Reca11，输入用户测试视频录像自动生成的转录文本，抽取测试者提及的可访问性与可用性问题。通过系统化的提示工程（prompt engineering），确定输入、指令、上下文与示例的最优组合配置。作者在三个应用、共 36 场用户测试会话上评估 Reca11 的效果，并据此分析将 LLM 用于该场景的优势与局限。
Testing for accessibility is a significant step when developing software, as it ensures that all users, including those with disabilities, can effectively engage with web and mobile applications. While automated tools exist to detect accessibility issues in software, none are as comprehensive and effective as the process of user testing, where testers with various disabilities evaluate the application for accessibility and usability issues. However, user testing is not popular with software developers as it requires conducting lengthy interviews with users and later parsing through large recordings to derive the issues to fix. In this paper, we explore how large language models (LLMs) like GPT 4.0, which have shown promising results in context comprehension and semantic text generation, can mitigate this issue and streamline the user testing process. Our solution, called Reca11, takes in auto-generated transcripts from user testing video recordings and extracts the accessibility and usability issues mentioned by the tester. Our systematic prompt engineering determines the optimal configuration of input, instruction, context and demonstrations for best results. We evaluate Reca11's effectiveness on 36 user testing sessions across three applications. Based on the findings, we investigate the strengths and weaknesses of using LLMs in this space.
DOI 原文 ·
@inproceedings{paperbot8927,
title = {Automated Generation of Accessibility Test Reports from Recorded User Transcripts},
author = {Syed Fatiul Huq and Mahan Tafreshipour and Kate Kalcevich and Sam Malek},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00043}
}