一句话:提出面向异步程序的新型测试充分性覆盖准则,涵盖异步操作的完成、异常、反应注册与执行,并实现工具 JScope 予以验证。
异步软件行为复杂且易错,而传统代码覆盖准则无法充分反映异步操作的完成情况、交互与错误处理。本文提出一组新的测试充分性准则,用于度量:(i) 异步操作在正常与异常两种执行路径下的完成;(ii) 为处理这两种可能结果而进行的反应(reaction)注册;(iii) 测试对这些反应的执行。作者实现了 JScope——一个面向 JavaScript 应用、以 Visual Studio Code 交互式插件形式提供的自动覆盖度量工具。在 20 个 JavaScript 应用上的评估表明,新准则能补充传统准则、改进测试充分性评估;对 15 个真实 GitHub 异步相关问题的调查显示,新准则能揭示那些虽被传统准则判定为已覆盖、实则未被测试的异步错误行为;此外,一项有 12 名参与者参加的受控实验表明,JScope 在真实场景中能有效提升程序员评估测试充分性以及检测异步代码未被测试行为的能力。
Asynchronous software often exhibits complex and error-prone behaviors that should be tested thoroughly. Code coverage has been the most popular metric to assess test suite quality. However, traditional code coverage criteria do not adequately reflect completion, interactions, and error handling of asynchronous operations. This paper proposes novel test adequacy criteria for measuring: (i) completion of asynchronous operations in terms of both successful and exceptional execution, (ii) registration of reactions for handling both possible outcomes, and (iii) execution of said reactions through tests. We implement JScope, a tool for automatically measuring coverage according to these criteria in JavaScript applications, as an interactive plug-in for Visual Studio Code. An evaluation of JScope on 20 JavaScript applications shows that the proposed criteria can help improve assessment of test adequacy, complementing traditional criteria. According to our investigation of 15 real GitHub issues concerned with asynchrony, the new criteria can help reveal faulty asynchronous behaviors that are untested yet are deemed covered by traditional coverage criteria. We also report on a controlled experiment with 12 participants to investigate the usefulness of JScope in realistic settings, demonstrating its effectiveness in improving programmers’ ability to assess test adequacy and detect untested behavior of asynchronous code.