一句话:综述学术界与工业界应对 flaky test 的方法,指出研究集中于单元测试而工业工具更关注系统级测试。
测试不稳定性(test flakiness)是软件自动化测试中的常见问题,会削弱测试自动化的有效性与生产力。本文对软件研究社区近期的相关方法以及主流工业测试自动化工具中处理测试不稳定性的能力进行了综述。作者从单元测试到端到端系统测试的范围内梳理现有工作,比较了学术研究与工业实践在关注点上的差异。综述发现,大量研究聚焦于单元测试层面的测试不稳定性问题,却少有工作处理端到端系统测试中的不稳定性;相反,工业测试自动化工具往往更关心系统层面的测试不稳定性。该工作揭示了学术研究与工业需求之间的错位,为后续面向系统级测试不稳定性研究指明了方向。
Test flakiness has been a common problem in the software automation testing, affecting the effectiveness and productivity of test automation. Many studies have been published to tackle test flakiness in the software research community, and existing test automation tools provide capabilities to address issues related to test flakiness. This paper describes our review of recent approaches in the research community and popular industrial tools with capabilities to tackle test flakiness. Our review indicates that while many studies focus on test flakiness in unit testing, few address the problem in the end-to-end system testing. On the contrary, industrial test automation tools tend to be more concerned with test flakiness at the system level.