一句话:该论文设计并评估了在 Amazon Mechanical Turk 上进行的众包 Planning Poker,证明精心组织的众包可达到接近单个专家的估算精度。
软件项目中的工作量估算长期面临可靠性与成本效益的挑战。Planning Poker 在具备领域知识的团队内可产生可靠估算,但依赖专家且耗时,难以用于可能有数千待处理任务的大型或开源项目。本文研究让仅获得有限任务信息的众包工人通过 Planning Poker 给出可比准确度估算的可行性。作者设计了在 Amazon Mechanical Turk 上运行的 Crowd Planning Poker 流程,并开展了涉及 5000 多名众包工人和 39 个不同软件任务的大规模试验。结果表明,经过精心组织和筛选的众包工人能够给出与单个专家准确度相似的工作量估算。
Reliable cost effective effort estimation remains a considerable challenge for software projects. Recent work has demonstrated that the popular Planning Poker practice can produce reliable estimates when undertaken within a software team of knowledgeable domain experts. However, the process depends on the availability of experts and can be time-consuming to perform, making it impractical for large scale or open source projects that may curate many thousands of outstanding tasks. This paper reports on a full study to investigate the feasibility of using crowd workers supplied with limited information about a task to provide comparably accurate estimates using Planning Poker. We describe the design of a Crowd Planning Poker (CPP) process implemented on Amazon Mechanical Turk and the results of a substantial set of trials, involving more than 5000 crowd workers and 39 diverse software tasks. Our results show that a carefully organised and selected crowd of workers can produce effort estimates that are of similar accuracy to those of a single expert.