paperbot · PL 论文追踪

全部论文 待读 收藏

P2 On the Reproducibility of Software Defect Datasets

论软件缺陷数据集的可复现性
ICSE 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)2023匹配度 2/10 · 该论文研究软件缺陷数据集的可复现性,属于经验软件工程与数据集维护方向,虽与'为评测提供可靠 benchmark'的兴趣有微弱交集,但不涉及静态分析、LLM/Agentic 程序分析、浮点或贝叶斯程序分析,与核心研究问题基本无关。引用 16 inbox
Hao-Nan Zhu, Cindy Rubio-González
crossref: 2026-09-12dblp: 2026-09-12

AI 速览

一句话:研究软件缺陷数据集的可复现性,发现广泛失效并提出依赖缓存与工件隔离,将复现率提升至95%以上。

软件缺陷数据集对故障定位、测试生成与自动程序修复等技术的评估和比较至关重要,但其工件易因环境变化而失效。本文研究其可复现性:先考察五个先进 Java 缺陷数据集,发现即便采用多种维护策略,仍普遍可能失效。随后进行13个月案例研究,系统测试1,795个软件工件,62.6%至少失效一次,15.3%多次失效。作者人工分析根因并编写10个补丁,自动应用到1,055个不同工件、共2,948次修复;据此提出自动依赖缓存与工件隔离。隔离消除外部依赖后,可复现性达95%以上,接近最可靠人工整理数据集。

问题
软件缺陷数据集是故障定位、测试生成和自动程序修复等评估的基础,但其工件可复现性可能随环境变化而失效,现有维护策略未必足够。
方法
对五个先进 Java 缺陷数据集进行考察,并开展为期13个月的案例研究,系统测试1,795个工件;人工分析失效根因、编写补丁并自动修复,提出依赖缓存与工件隔离。
评估
统计工件失效比例与修复效果,并评估依赖缓存和工件隔离对可复现性的提升;隔离后复现率达95%以上,接近最可靠人工数据集。
原文摘要(Abstract)

Software defect datasets are crucial to facilitating the evaluation and comparison of techniques in fields such as fault localization, test generation, and automated program repair. However, the reproducibility of software defect artifacts is not immune to breakage. In this paper, we conduct a study on the reproducibility of software defect artifacts. First, we study five state-of-the-art Java defect datasets. Despite the multiple strategies applied by dataset maintainers to ensure reproducibility, all datasets are prone to breakages. Second, we conduct a case study in which we systematically test the reproducibility of 1,795 software artifacts during a 13-month period. We find that 62.6% of the artifacts break at least once, and 15.3% artifacts break multiple times. We manually investigate the root causes of breakages and handcraft 10 patches, which are automatically applied to 1,055 distinct artifacts in 2,948 fixes. Based on the nature of the root causes, we propose automated dependency caching and artifact isolation to prevent further breakage. In particular, we show that isolating artifacts to eliminate external dependencies increases reproducibility to 95% or higher, which is on par with the level of reproducibility exhibited by the most reliable manually curated dataset.

链接与引用

DOI 原文 · DBLP

BibTeX
@inproceedings{ZhuR23,
  title = {On the Reproducibility of Software Defect Datasets},
  author = {Hao-Nan Zhu and Cindy Rubio-González},
  booktitle = {ICSE},
  year = {2023},
  doi = {10.1109/icse48619.2023.00195}
}