一句话：通过众包收集软件工程师对法规歧义的标注，并用 EM 达成共识，以 87% 准确率匹配专家标注。
合规软件对企业至关重要，而从法规中推导需求必须先消解其歧义。法规文本充满歧义，通常依赖法律专家理解，但让专家参与每个项目成本高昂。自动化消歧方法需要大量标注数据，仅靠专家标注难以扩展且负担重。本文开展众包实验，让专业软件工程师标注法规中的歧义，并研究如何自动确定 ground truth 标签。作者采用期望最大化（EM）方法聚合众包标注，通过群体共识得到可靠标签。实验表明，达成共识的标注与专家标注的准确率约为 87%。该工作为法规消歧所需的标注数据获取与质量控制提供了一条可扩展路径。
Compliant software is a critical need of all modern businesses. Disambiguating regulations to derive requirements is therefore an important software engineering activity. Regulations however are ridden with ambiguities that make their comprehension a challenge, seemingly surmountable only by legal experts. Since legal experts' involvement in every project is expensive, approaches to automate the disambiguation need to be explored. These approaches however require a large amount of annotated data. Collecting data exclusively from experts is not a scalable and affordable solution. In this paper, we present the results of a crowd sourcing experiment to collect annotations on ambiguities in regulations from professional software engineers. We discuss an approach to automate the arduous and critical step of identifying ground truth labels by employing crowd consensus using Expectation Maximization (EM). We demonstrate that the annotations reaching a consensus match those of experts with an accuracy of 87%.
DOI 原文 ·
@inproceedings{paperbot4770,
title = {Towards automating disambiguation of regulations: using the wisdom of crowds},
author = {Manasi Patwardhan and Abhishek Sainani and Richa Sharma and Shirish Karande and Smita Ghaisas},
booktitle = {ASE},
year = {2018},
doi = {10.1145/3238147.3240727}
}