一句话：提出 FASER，通过优化断言边界来权衡非确定性机器学习测试的故障检测有效性与不稳定率。
机器学习项目的测试面临算法固有非确定性和缺乏可靠参考结果的问题。开发者通常凭直觉设置测试断言中实际值与期望值的比较边界，并为了避免误报而过于保守，导致遗漏关键缺陷。本文提出 FASER，首个系统性地平衡非确定性测试故障检测有效性与不稳定率的方法，将断言边界的选择建模为两个竞争目标之间的优化问题。FASER 使用统计方法估计不稳定率，并使用变异测试估计故障检测有效性，从而计算最优断言边界。作者在来自 22 个流行机器学习项目的 87 个非确定性测试上评估 FASER，发现其中 23 个测试使用了保守边界，并给出了在限制不稳定率的同时最大化故障检测有效性的更紧断言边界。作者提交了 19 个修复测试的 pull request，其中 14 个已被接受。
Testing Machine Learning (ML) projects is challenging due to inherent non-determinism of various ML algorithms and the lack of reliable ways to compute reference results. Developers typically rely on their intuition when writing tests to check whether ML algorithms produce accurate results. However, this approach leads to conservative choices in selecting assertion bounds for comparing actual and expected results in test assertions. Because developers want to avoid false positive failures in tests, they often set the bounds to be too loose, potentially leading to missing critical bugs. We present FASER - the first systematic approach for balancing the trade-off between the fault-detection effectiveness and flakiness of non-deterministic tests by computing optimal assertion bounds. FASER frames this trade-off as an optimization problem between these competing objectives by varying the assertion bound. FASER leverages 1) statistical methods to estimate the flakiness rate, and 2) mutation testing to estimate the fault-detection effectiveness. We evaluate FASER on 87 non-deterministic tests collected from 22 popular ML projects. FASER finds that 23 out of 87 studied tests have conservative bounds and proposes tighter assertion bounds that maximizes the fault-detection effectiveness of the tests while limiting flakiness. We have sent 19 pull requests to developers, each fixing one test, out of which 14 pull requests have already been accepted.
@inproceedings{XiaDMMZ23,
title = {Balancing Effectiveness and Flakiness of Non-Deterministic Machine Learning Tests},
author = {Chunqiu Steven Xia and Saikat Dutta and Sasa Misailovic and Darko Marinov and Lingming Zhang},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00154}
}