一句话:研究软件缺陷数据集的可复现性,发现广泛失效并提出依赖缓存与工件隔离,将复现率提升至95%以上。
软件缺陷数据集对故障定位、测试生成与自动程序修复等技术的评估和比较至关重要,但其工件易因环境变化而失效。本文研究其可复现性:先考察五个先进 Java 缺陷数据集,发现即便采用多种维护策略,仍普遍可能失效。随后进行13个月案例研究,系统测试1,795个软件工件,62.6%至少失效一次,15.3%多次失效。作者人工分析根因并编写10个补丁,自动应用到1,055个不同工件、共2,948次修复;据此提出自动依赖缓存与工件隔离。隔离消除外部依赖后,可复现性达95%以上,接近最可靠人工整理数据集。
Software defect datasets are crucial to facilitating the evaluation and comparison of techniques in fields such as fault localization, test generation, and automated program repair. However, the reproducibility of software defect artifacts is not immune to breakage. In this paper, we conduct a study on the reproducibility of software defect artifacts. First, we study five state-of-the-art Java defect datasets. Despite the multiple strategies applied by dataset maintainers to ensure reproducibility, all datasets are prone to breakages. Second, we conduct a case study in which we systematically test the reproducibility of 1,795 software artifacts during a 13-month period. We find that 62.6% of the artifacts break at least once, and 15.3% artifacts break multiple times. We manually investigate the root causes of breakages and handcraft 10 patches, which are automatically applied to 1,055 distinct artifacts in 2,948 fixes. Based on the nature of the root causes, we propose automated dependency caching and artifact isolation to prevent further breakage. In particular, we show that isolating artifacts to eliminate external dependencies increases reproducibility to 95% or higher, which is on par with the level of reproducibility exhibited by the most reliable manually curated dataset.