一句话：系统评估 6 款 Java 质量保证工具，分析规则覆盖、缺陷检测效果与时间性能，总结工具改进和选型启示。
质量保证（QA）工具被广泛使用，但现有评估常忽略扫描规则分析，因方法与基准不同导致结论不一致，未单独分析警告的作用，也缺少大规模时间性能研究。本文从 148 款 Java QA 工具中筛选 6 款免费或开源工具，将扫描规则映射到 CWE，分析其覆盖范围与粒度；在包含 1,425 个 bug 的 5 个基准上开展实验，对比真实标注 bug 与工具警告以考察警告在缺陷检测中的作用；并在 1,049 个项目上测量时间性能。研究给出多层级维度的实证发现，可帮助开发者改进工具，并为用户选择 QA 工具提供建议。
Quality assurance (QA) tools are receiving more and more attention and are widely used by developers. Given the wide range of solutions for QA technology, it is still a question of evaluating QA tools. Most existing research is limited in the following ways: (i) They compare tools without considering scanning rules analysis. (ii) They disagree on the effectiveness of tools due to the study methodology and benchmark dataset. (iii) They do not separately analyze the role of the warnings. (iv) There is no large-scale study on the analysis of time performance. To address these problems, in the paper, we systematically select 6 free or open-source tools for a comprehensive study from a list of 148 existing Java QA tools. To carry out a comprehensive study and evaluate tools in multi-level dimensions, we first mapped the scanning rules to the CWE and analyze the coverage and granularity of the scanning rules. Then we conducted an experiment on 5 benchmarks, including 1,425 bugs, to investigate the effectiveness of these tools. Furthermore, we took substantial effort to investigate the effectiveness of warnings by comparing the real labeled bugs with the warnings and investigating their role in bug detection. Finally, we assessed these tools’ time performance on 1,049 projects. The useful findings based on our comprehensive study can help developers improve their tools and provide users with suggestions for selecting QA tools.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{LiuCFLLXNLC23,
title = {A Comprehensive Study on Quality Assurance Tools for Java},
author = {Han Liu and Sen Chen and Ruitao Feng and Chengwei Liu and Kaixuan Li and Zhengzi Xu and Liming Nie and Yang Liu and Yixiang Chen},
booktitle = {ISSTA},
year = {2023},
doi = {10.1145/3597926.3598056}
}