一句话：通过模拟批量测试的批大小与 flaky test 影响，并用缺陷/执行历史模型引导二分，降低工业测试成本。
软件测试成本高昂。批量测试虽能减少初始测试执行，但批内失败需二分查找 culprit commit；批越大初始运行越少，却更易触发昂贵重测。本文基于 Ericsson 三个大型工业系统 9 个月测试数据开展实证与模拟：将批大小从 1 到 20 模拟，发现批量测试相比逐提交测试节省 72% 执行，并确定各项目最经济的 BatchSize；引入 flaky tests 后节省降至 42%。进一步用缺陷模型和测试执行历史模型评估提交风险，高风险提交单独测试、低风险提交合批，以引导二分。相比 Ericsson 现有二分方法，culprit 预测最多减少 9% 测试执行；结果已被采用，工具正加入 CI 流水线。
Software testing is one of the costliest stages in the software development life cycle. One approach to reducing the test execution cost is to group changes and test them as a batch (i.e. batch testing). However, when tests fail in a batch, commits in the batch need to be re-tested to identify the cause of the failure, i.e. the culprit commit. The re-testing is typically done through bisection (i.e. a binary search through the commits in a batch). Intuitively, the effectiveness of batch testing highly depends on the size of the batch. Larger batches require fewer initial test runs, but have a higher chance of a test failure that can lead to expensive test re-runs to find the culprit. We are unaware of research that investigates and simulates the impact of batch sizes on the cost of testing in industry. In this work, we first conduct empirical studies on the effectiveness of batch testing in three large-scale industrial software systems at Ericsson. Using 9 months of testing data, we simulate batch sizes from 1 to 20 and find the most cost-effective BatchSize for each project. Our results show that batch testing saves 72% of test executions compared to testing each commit individually. In a second simulation, we incorporate flaky tests that pass and fail on the same commit as they are a significant source of additional test executions on large projects. We model the degree of flakiness for each project and find that test flakiness reduces the cost savings to 42%. In a third simulation, we guide bisection to reduce the likelihood of batch-testing failures. We model the riskiness of each commit in a batch using a bug model and a test execution history model. The risky commits are tested individually, while the less risky commits are tested in a single larger batch. Culprit predictions with our approach reduce test executions up to 9% compared to Ericsson’s current bisection approach. The results have been adopted by developers at Ericsson and a tool to guide bisection is in the process of being added to Ericsson’s continuous integration pipeline.
DOI 原文 ·
@inproceedings{paperbot5059,
title = {Bisecting commits and modeling commit risk during testing},
author = {Armin Najafi and Peter C. Rigby and Weiyi Shang},
booktitle = {FSE},
year = {2019},
doi = {10.1145/3338906.3338944}
}