一句话：通过问卷调查研究软件工程与编程语言社区对研究工件评估的期望、共识与术语不一致问题。
该论文关注软件工程与编程语言会议中研究工件评估（artifact evaluation）的实践。作者针对自 2011 年 ESEC/FSE 试点以来各主要会议工件评估委员会成员开展问卷调查，并将其回答与工件征集说明及评审指南中的期望进行对比。结果发现：社区的部分期望超出了征集说明所表达的内容；对工件质量阈值缺乏普遍共识；针对不同类型工件在评审与后续使用上存在非常具体的质量期望，但这些期望在征集说明中未被充分传达；此外，用于表达工件评估最重要目的——可复现性（replicability）——的术语存在不一致，构成问题。基于此，作者提出若干可操作建议，以促进该社区工件评估的成熟，并帮助其他计算机科学社区引入类似实践。
Background. Artifact evaluation has been introduced into the software engineering and programming languages research community with a pilot at ESEC/FSE 2011 and has since then enjoyed a healthy adoption throughout the conference landscape. Objective. In this qualitative study, we examine the expectations of the community toward research artifacts and their evaluation processes. Method. We conducted a survey including all members of artifact evaluation committees of major conferences in the software engineering and programming language field since the first pilot and compared the answers to expectations set by calls for artifacts and reviewing guidelines. Results. While we find that some expectations exceed the ones expressed in calls and reviewing guidelines, there is no consensus on quality thresholds for artifacts in general. We observe very specific quality expectations for specific artifact types for review and later usage, but also a lack of their communication in calls. We also find problematic inconsistencies in the terminology used to express artifact evaluation’s most important purpose – replicability. Conclusion. We derive several actionable suggestions which can help to mature artifact evaluation in the inspected community and also to aid its introduction into other communities in computer science.
DOI 原文 ·
@inproceedings{paperbot5820,
title = {Community expectations for research artifacts and evaluation processes},
author = {Ben Hermann and Stefan Winter and Janet Siegmund},
booktitle = {FSE},
year = {2020},
doi = {10.1145/3368089.3409767}
}