一句话:揭示 IRFL 中 MAP 度量实现的常见偏差及其导致的最高达 70% 的性能高估,并主张严格遵循教科书定义。
MAP(Mean Average Precision)是信息检索故障定位(IRFL)中最常用的性能指标之一,但研究中存在偏离教科书定义的实现。这些实现使指标对检索结果截断、数据集不精确与杂质敏感,可能导致性能高估,影响 IRFL 结果的可比性、可迁移性与有效性。本文讨论 MAP 的定义与数学性质,分析常见偏差与实现陷阱,识别出导致高估的条件:检索结果截断与跨多文件的 ground truth 结合,以及未定义 AP 值的不当处理。作者在 Bench4BL 基准上使用五种知名 IRFL 技术演示了高估效应,极端情况下高估达 70%。论文主张 IRFL 实验严格遵循教科书版 MAP,并将未定义 AP 值设为 0,以提高研究的可比性与可迁移性。
The MAP (Mean Average Precision) metric is one of the most popular performance metrics in the field of Information Retrieval Fault Localization (IRFL). However, there are problematic implementations of this MAP metric used in IRFL research. These implementations deviate from the text book definitions of MAP, rendering the metric sensitive to the truncation of retrieval results and inaccuracies and impurities of the used datasets. The application of such a deviating metric can lead to performance overestimation. This can pose a problem for comparability, transferability, and validity of IRFL performance results. In this paper, we discuss the definition and mathematical properties of MAP and common deviations and pitfalls in its implementation. We investigate and discuss the conditions enabling such overestimation: the truncation of retrieval results in combination with ground truths spanning multiple files and improper handling of undefined AP results. We demonstrate the overestimation effects using the Bench4BL benchmark and five well known IRFL techniques. Our results indicate that a flawed implementation of the MAP metric can lead to an overestimation of the IRFL performance, in extreme cases by up to 70 %. We argue for a strict adherence to the text book version of MAP with the extension of undefined AP values to be set to 0 for all IRFL experiments. We hope that this work will help to improve comparability and transferability in IRFL research.