paperbot · PL 论文追踪

全部论文 待读 收藏

P2 The MAP Metric in Information Retrieval Fault Localization

信息检索故障定位中的 MAP 度量
ASE 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)2023匹配度 3/10 · 该论文讨论信息检索式故障定位(IRFL)中 MAP 度量的实现偏差与性能高估问题,属于软件测试/调试评估方法,与用户核心关注的静态分析、LLM/Agent 程序分析、浮点程序分析及贝叶斯程序分析均无直接关系;仅在‘程序分析/调试工具评估严谨性’这一较宽泛意义上有弱关联。引用 5 inbox
Thomas Hirsch, Birgit Hofer
crossref: 2026-09-12dblp: 2026-09-12

AI 速览

一句话:揭示 IRFL 中 MAP 度量实现的常见偏差及其导致的最高达 70% 的性能高估,并主张严格遵循教科书定义。

MAP(Mean Average Precision)是信息检索故障定位(IRFL)中最常用的性能指标之一,但研究中存在偏离教科书定义的实现。这些实现使指标对检索结果截断、数据集不精确与杂质敏感,可能导致性能高估,影响 IRFL 结果的可比性、可迁移性与有效性。本文讨论 MAP 的定义与数学性质,分析常见偏差与实现陷阱,识别出导致高估的条件:检索结果截断与跨多文件的 ground truth 结合,以及未定义 AP 值的不当处理。作者在 Bench4BL 基准上使用五种知名 IRFL 技术演示了高估效应,极端情况下高估达 70%。论文主张 IRFL 实验严格遵循教科书版 MAP,并将未定义 AP 值设为 0,以提高研究的可比性与可迁移性。

问题
IRFL 研究中普遍使用的 MAP 度量实现常偏离教科书定义,导致指标对检索结果截断和数据集不精确敏感,从而产生性能高估,损害结果的可比性与有效性。
方法
通过数学分析讨论 MAP 的定义与性质,剖析常见实现偏差与陷阱,归纳出导致高估的两个条件:检索结果截断与跨文件 ground truth 组合,以及未定义 AP 值处理不当。
评估
在 Bench4BL 基准上使用五种知名 IRFL 技术进行实证,发现缺陷实现可导致 IRFL 性能高估,极端情况下高达 70%。
原文摘要(Abstract)

The MAP (Mean Average Precision) metric is one of the most popular performance metrics in the field of Information Retrieval Fault Localization (IRFL). However, there are problematic implementations of this MAP metric used in IRFL research. These implementations deviate from the text book definitions of MAP, rendering the metric sensitive to the truncation of retrieval results and inaccuracies and impurities of the used datasets. The application of such a deviating metric can lead to performance overestimation. This can pose a problem for comparability, transferability, and validity of IRFL performance results. In this paper, we discuss the definition and mathematical properties of MAP and common deviations and pitfalls in its implementation. We investigate and discuss the conditions enabling such overestimation: the truncation of retrieval results in combination with ground truths spanning multiple files and improper handling of undefined AP results. We demonstrate the overestimation effects using the Bench4BL benchmark and five well known IRFL techniques. Our results indicate that a flawed implementation of the MAP metric can lead to an overestimation of the IRFL performance, in extreme cases by up to 70 %. We argue for a strict adherence to the text book version of MAP with the extension of undefined AP values to be set to 0 for all IRFL experiments. We hope that this work will help to improve comparability and transferability in IRFL research.

链接与引用

DOI 原文 · PDF(开放获取) · DBLP

BibTeX
@inproceedings{HirschH23,
  title = {The MAP Metric in Information Retrieval Fault Localization},
  author = {Thomas Hirsch and Birgit Hofer},
  booktitle = {ASE},
  year = {2023},
  doi = {10.1109/ase56229.2023.00041}
}