一句话：提出利用作业遥测数据训练分类模型，自动区分 Mozilla CI 中间歇性失败与真实软件缺陷。
持续交付的云系统需要不断运行构建、测试等作业，但 CI 中会出现间歇性失败（Mozilla 称为 oranges）：它们并非由软件或作业规格缺陷造成，而是源于基础设施等问题的非确定性误报，会扰乱 CI 并削弱开发者对作业的信任。本文提出自动分类失败作业的方法，判断失败来自真实软件 bug，还是作业/测试或底层基础设施的 flakiness。方法利用作业遥测数据训练分类模型，结合运行时、CPU 负载、操作系统版本、特定平台等特征诊断失败模式。在 Mozilla CI 作业集上评估，平均精度 73%，部分测试套件精度可达 100% 并足以全自动分类，平均召回率 82%，最高 94%。
Continuous delivery of cloud systems requires constant running of jobs (build processes, tests, etc.). One issue that plagues this continuous integration (CI) process are intermittent failures - non-deterministic, false alarms that do not result from a bug in the software or job specification, but rather from issues in the underlying infrastructure. At Mozilla, such intermittent failures are called oranges as a reference to the color of the build status indicator. As such intermittent failures disrupt CI and lead to failures, they erode the developers' trust in the jobs. We present a novel approach that automatically classifies failing jobs to determine whether job execution failures arise from an actual software bug or were caused by flakiness in the job (e.g., test) or the underlying infrastructure. For this purpose, we train classification models using job telemetry data to diagnose failure patterns involving features such as runtime, cpu load, operating system version, or specific platform with high precision. In an evaluation on a set of Mozilla CI jobs, our approach achieves precision scores of 73%, on average, across all data sets with some test suites achieving precision scores good enough for fully automated classification (i.e., precision scores of up to 100%), and recall scores of 82% on average (up to 94%).
@inproceedings{paperbot6392,
title = {When life gives you oranges: detecting and diagnosing intermittent job failures at Mozilla},
author = {Johannes Lampel and Sascha Just and Sven Apel and Andreas Zeller},
booktitle = {FSE},
year = {2021},
doi = {10.1145/3468264.3473931}
}