一句话：提出整体性功能测试与缺陷类别方法，检测机器学习信息检索软件的系统级回归与改进。
针对基于机器学习的信息检索软件，传统测试难以从单个用例评估系统级涌现行为与大规模回归/改进。本文提出整体性功能测试方法，利用自动获取的大规模功能测试用例，从终端用户视角度量系统行为，并引入“缺陷类别”概念——对输入空间进行划分，标识现有特征表现明显变差或新特征更具挑战的区域。在包含查询理解机器学习组件的真实 Search-AutoComplete 生产系统上评估：整体度量成功识别两个回归和一个改进，三者均经回顾性 A/B 实验独立验证；自动缺陷类别在早期机器学习开发中提供可操作洞见；并在更细粒度子组件级检测到显著回归，从而在发布前阻止。
Machine Learning (ML) has become the cornerstone of information retrieval (IR) software, as it can drive better user experience by leveraging information-rich data and complex models. However, evaluating the emergent behavior of ML-based IR software can be challenging with traditional software testing approaches: when developers modify the software, they cannot often extract useful information from individual test instances; rather, they seek to holistically verify whether—and where—their modifications caused significant regressions or improvements at scale. In this paper, we introduce not only such a holistic approach to evaluate the system-level behavior of the software, but also the concept of a defect class, which represents a partition of the input space on which the ML-based software does measurably worse for an existing feature or on which the ML task is more challenging for a new feature. We leverage large volumes of functional test cases, automatically obtained, to derive these defect classes, and propose new ways to improve the IR software from an end-user’s perspective. Applying our approach on a real production Search-AutoComplete system that contains a query interpretation ML component, we demonstrate that (1) our holistic metrics successfully identified two regressions and one improvement, where all 3 were independently verified with retrospective A/B experiments, (2) the automatically obtained defect classes provided actionable insights during early-stage ML development, and (3) we also detected defect classes at the finer sub-component level for which there were significant regressions, which we blocked prior to different releases.
DOI 原文 ·
@inproceedings{paperbot7126,
title = {Improving ML-based information retrieval software with user-driven functional testing and defect class analysis},
author = {Junjie Zhu and Teng Long and Wei Wang and Atif Memon},
booktitle = {FSE},
year = {2022},
doi = {10.1145/3540250.3558941}
}