一句话：工业案例研究评估 EvoMaster 对挪威癌症登记处医疗规则引擎 GURI 的测试效果，发现黑盒工具在领域特定规则覆盖上最优。
挪威癌症登记处（CRN）的决策支持系统需要自动化测试，但当前实践有限。本文开展工业案例研究，评估 AI 驱动的系统级测试工具 EvoMaster 对 CRN 关键组件 GURI（医疗规则引擎）的测试效果。方法上，使用 EvoMaster 的黑盒与白盒工具，从代码覆盖率、错误发现和领域特定规则覆盖三方面对比。结果表明，各工具代码覆盖率相近（约19%行、13%分支、20%方法），均发现1个错误；黑盒工具在领域特定覆盖上最优，能触发100%聚合规则和12.86%–25.81%验证规则，产生多样的规则执行结果，且在10个规则版本上结果一致。作者建议采用黑盒工具，但需扩展领域特定优化目标以进一步提升效果。
The Cancer Registry of Norway (CRN) collects, curates, and manages data related to cancer patients in Norway, supported by an interactive, human-in-the-loop, socio-technical decision support software system. Automated software testing of this software system is inevitable; however, currently, it is limited in CRN’s practice. To this end, we present an industrial case study to evaluate an AI-based system-level testing tool, i.e., EvoMaster, in terms of its effectiveness in testing CRN’s software system. In particular, we focus on GURI, CRN’s medical rule engine, which is a key component at the CRN. We test GURI with EvoMaster’s black-box and white-box tools and study their test effectiveness regarding code coverage, errors found, and domain-specific rule coverage. The results show that all EvoMaster tools achieve a similar code coverage; i.e., around 19% line, 13% branch, and 20% method; and find a similar number of errors; i.e., 1 in GURI’s code. Concerning domain-specific coverage, EvoMaster’s black-box tool is the most effective in generating tests that lead to applied rules; i.e., 100% of the aggregation rules and between 12.86% and 25.81% of the validation rules; and to diverse rule execution results; i.e., 86.84% to 89.95% of the aggregation rules and 0.93% to 1.72% of the validation rules pass, and 1.70% to 3.12% of the aggregation rules and 1.58% to 3.74% of the validation rules fail. We further observe that the results are consistent across 10 versions of the rules. Based on these results, we recommend using EvoMaster’s black-box tool to test GURI since it provides good results and advances the current state of practice at the CRN. Nonetheless, EvoMaster needs to be extended to employ domain-specific optimization objectives to improve test effectiveness further. Finally, we conclude with lessons learned and potential research directions, which we believe are applicable in a general context.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{Laaber00SN23,
title = {Automated Test Generation for Medical Rules Web Services: A Case Study at the Cancer Registry of Norway},
author = {Christoph Laaber and Tao Yue and Shaukat Ali and Thomas Schwitalla and Jan F. Nygård},
booktitle = {FSE},
year = {2023},
doi = {10.1145/3611643.3613882}
}