paperbot · PL 论文追踪

全部论文 待读 收藏

P2 NaNofuzz: A Usable Tool for Automatic Test Generation

NaNofuzz:一个可用的自动测试生成工具
FSE Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering2023匹配度 2/10 · 论文聚焦自动测试生成工具的可用性与随机对照实验,属于软件测试/开发者工具方向,未涉及静态分析、LLM/Agent、形式化验证、浮点程序分析或贝叶斯程序分析等用户核心研究方向。尽管标题和摘要提及 NaN 输出,但并非针对浮点程序分析或相关交叉问题。引用 23 inbox
Matthew C. Davis, Sangheon Choi, Sam Estep, Brad A. Myers, Joshua Sunshine
crossref: 2026-09-12dblp: 2026-09-12

AI 速览

一句话:提出嵌入开发环境的交互式测试生成工具 NaNofuzz,通过自动推断输入、执行并分类输出以优先暴露疑似缺陷,随机对照实验证明其提升效率与信心。

尽管测试执行与其他软件工程环节已高度自动化,测试用例的编写仍主要依靠人工,成本高昂。作者构建了交互式测试生成工具 NaNofuzz,它嵌入开发者日常使用的开发环境中:用户只需按一个按钮即可对某个函数发起测试,工具分析程序以确定应当尝试的输入,执行程序,并对输出进行分类,从而优先提示那些很可能指示缺陷的结果(例如返回 NaN 等异常值)。为评估其可用性与有效性,作者开展了一项随机对照实验,以 28 名职业软件工程师为被试,将 NaNofuzz 作为干预组、流行的人工测试工具 Jest 作为对照组。结果显示,使用 NaNofuzz 的被试识别缺陷的准确率平均提高 30%(p < .05),对测试的信心提升 20%(p < .03),并且完成任务的速度快 30%(p < .007),表明该工具在真实开发者工作流中兼具可用性与实际效益。

问题
测试用例的编写仍然依赖人工,成本极高(仅美国每年估计约 480 亿美元),而现有自动化测试生成工具往往难以融入开发者工作流、生成结果不易判定是否为真正缺陷。
方法
实现交互式测试生成工具 NaNofuzz,嵌入开发环境,一键即可对函数测试:分析程序以确定待评估的输入、执行这些输入并对输出分类,从而按可能指示缺陷的程度(如返回 NaN)排序提示。
评估
以 28 名职业软件工程师为被试开展随机对照实验,NaNofuzz 为干预组、人工测试工具 Jest 为对照组;结果显示使用 NaNofuzz 的被试识别缺陷更准确(提高 30%,p < .05)、对测试更有信心(提高 20%,p < .03)、完成任务更快(快 30%,p < .007)。
原文摘要(Abstract)

In the United States alone, software testing labor is estimated to cost $48 billion USD per year. Despite widespread test execution automation and automation in other areas of software engineering, test suites continue to be created manually by software engineers. We have built a test generation tool, called NaNofuzz, that helps users find bugs in their code by suggesting tests where the output is likely indicative of a bug, e.g., that return NaN (not-a-number) values. NaNofuzz is an interactive tool embedded in a development environment to fit into the programmer's workflow. NaNofuzz tests a function with as little as one button press, analyses the program to determine inputs it should evaluate, executes the program on those inputs, and categorizes outputs to prioritize likely bugs. We conducted a randomized controlled trial with 28 professional software engineers using NaNofuzz as the intervention treatment and the popular manual testing tool, Jest, as the control treatment. Participants using NaNofuzz on average identified bugs more accurately (p < .05, by 30%), were more confident in their tests (p < .03, by 20%), and finished their tasks more quickly (p < .007, by 30%).

链接与引用

DOI 原文 · PDF(开放获取) · DBLP

BibTeX
@inproceedings{DavisCEMS23,
  title = {NaNofuzz: A Usable Tool for Automatic Test Generation},
  author = {Matthew C. Davis and Sangheon Choi and Sam Estep and Brad A. Myers and Joshua Sunshine},
  booktitle = {FSE},
  year = {2023},
  doi = {10.1145/3611643.3616327}
}