一句话:回顾软件工程社区工件评估的兴起与运作,说明其如何提升研究结果的可信度。
过去数十年,多个科学社区出现可复现性危机:独立复现已发表结果失败,使结果有效性受到质疑。软件工程社区较早应对该问题,于 ESEC/FSE 2011 首次引入工件评估(artifact evaluation),作者自愿提交支撑论文结果的工件,接受同行评审。此后工件评估在软件工程会议中迅速普及,多数会议都提供该环节,并形成规模庞大的工件评估委员会来处理大量提交。部分会议还将工件评估与论文录用挂钩,要求工件通过评估论文才可发表。本文讨论这一实践如何增强研究结果的可靠性。
Over the past decades, reports of reproducibility crises have surfaced in various scientific communities. Independent confirmations of published research results failed, casting doubt on the validity of these results. Even before the magnitude of the problem has become apparent in many domains, the software-engineering community introduced artifact evaluations, for the first time at ESEC/FSE 2011, in which research artifacts that support published results were voluntarily submitted for peer review. Since then, artifact evaluations have become immensely popular and are today being offered to authors at most software-engineering venues, where large artifact-evaluation committees handle large numbers of artifact submissions. At some venues, papers are accepted for publication only if their artifacts pass the artifact evaluation.