一句话：提出 FlakeRepro，结合静态与动态分析高效复现并发导致的不稳定测试失败，探索数十个交错即可复现相同错误。
Flaky tests 会在同一代码上非确定性地通过或失败，在回归测试中给出误导信号，显著拖慢开发。多数单元测试的不稳定性由并发引起，而复现其失败需要特定的线程交错，这并非易事。FlakeRepro 旨在帮助开发者复现并发相关 flaky test 的失败执行。它结合静态分析与动态分析，快速识别出能使测试以原始错误信息失败的线程交错。作者报告 FlakeRepro 效率较高，探索数十个交错即可复现失败执行；同时它能自动插桩测试二进制，从而在现有且未修改的测试流水线中运行。
Flaky tests, which can non-deterministically pass or fail on the same code, impose significant burden on developers by providing misleading signals during regression testing. Microsoft developers consider flaky tests as one of the top two reasons for slowing down software development. In order to debug the root-cause of a flaky behavior, a developer often needs to first reliably reproduce a failed execution. Unfortunately, this is non-trivial. For example, most of the flakiness in unit tests are caused by concurrency, and reproducing their failures requires specific thread interleaving. To address this challenge, we introduce FlakeRepro that helps developers reproduce a failed execution of a flaky test caused by concurrency. FlakeRepro combines static and dynamic analysis to quickly identify an interleaving that makes a flaky test fail with the same original error message. FlakeRepro is efficient: it can reproduce a failed execution after exploring few tens of interleavings. FlakeRepro integrates well with existing systems: it automatically instruments test binaries that can run on existing and unmodified test pipelines.
DOI 原文 ·
@inproceedings{paperbot7093,
title = {FlakeRepro: automated and efficient reproduction of concurrency-related flaky tests},
author = {Tanakorn Leesatapornwongsa and Xiang Ren and Suman Nath},
booktitle = {FSE},
year = {2022},
doi = {10.1145/3540250.3558956}
}