一句话:提出嵌入开发环境的交互式测试生成工具 NaNofuzz,通过自动推断输入、执行并分类输出以优先暴露疑似缺陷,随机对照实验证明其提升效率与信心。
尽管测试执行与其他软件工程环节已高度自动化,测试用例的编写仍主要依靠人工,成本高昂。作者构建了交互式测试生成工具 NaNofuzz,它嵌入开发者日常使用的开发环境中:用户只需按一个按钮即可对某个函数发起测试,工具分析程序以确定应当尝试的输入,执行程序,并对输出进行分类,从而优先提示那些很可能指示缺陷的结果(例如返回 NaN 等异常值)。为评估其可用性与有效性,作者开展了一项随机对照实验,以 28 名职业软件工程师为被试,将 NaNofuzz 作为干预组、流行的人工测试工具 Jest 作为对照组。结果显示,使用 NaNofuzz 的被试识别缺陷的准确率平均提高 30%(p < .05),对测试的信心提升 20%(p < .03),并且完成任务的速度快 30%(p < .007),表明该工具在真实开发者工作流中兼具可用性与实际效益。
In the United States alone, software testing labor is estimated to cost $48 billion USD per year. Despite widespread test execution automation and automation in other areas of software engineering, test suites continue to be created manually by software engineers. We have built a test generation tool, called NaNofuzz, that helps users find bugs in their code by suggesting tests where the output is likely indicative of a bug, e.g., that return NaN (not-a-number) values. NaNofuzz is an interactive tool embedded in a development environment to fit into the programmer's workflow. NaNofuzz tests a function with as little as one button press, analyses the program to determine inputs it should evaluate, executes the program on those inputs, and categorizes outputs to prioritize likely bugs. We conducted a randomized controlled trial with 28 professional software engineers using NaNofuzz as the intervention treatment and the popular manual testing tool, Jest, as the control treatment. Participants using NaNofuzz on average identified bugs more accurately (p < .05, by 30%), were more confident in their tests (p < .03, by 20%), and finished their tasks more quickly (p < .007, by 30%).