一句话：系统比较七个开源 Java SAST 工具，揭示其在真实漏洞上召回率低且互补性有限。
静态应用安全测试（SAST）在软件开发生命周期中至关重要，但全面评估 SAST 工具的有效性仍具挑战。本文基于明确标准，从161个现有工具中选出七个免费或开源 SAST 工具，利用合成基准与新构建的真实世界基准，从有效性、一致性和性能等维度进行比较评估。结果显示，这些工具在合成基准上表现良好，但仅能检测出12.7%的真实世界漏洞；即使合并所有工具的检测能力，仍有70.9%的漏洞未被发现，尤其是资源控制类以及输入/输出中和不充分类漏洞。尽管已构建相应检测规则，检测结果仍不符合预期。研究为工具开发、改进、评估与选择提供指导。
Static application security testing (SAST) takes a significant role in the software development life cycle (SDLC). However, it is challenging to comprehensively evaluate the effectiveness of SAST tools to determine which is the better one for detecting vulnerabilities. In this paper, based on well-defined criteria, we first selected seven free or open-source SAST tools from 161 existing tools for further evaluation. Owing to the synthetic and newly-constructed real-world benchmarks, we evaluated and compared these SAST tools from different and comprehensive perspectives such as effectiveness, consistency, and performance. While SAST tools perform well on synthetic benchmarks, our results indicate that only 12.7% of real-world vulnerabilities can be detected by the selected tools. Even combining the detection capability of all tools, most vulnerabilities (70.9%) remain undetected, especially those beyond resource control and insufficiently neutralized input/output vulnerabilities. The fact is that although they have already built the corresponding detecting rules and integrated them into their capabilities, the detection result still did not meet the expectations. All useful findings unveiled in our comprehensive study indeed help to provide guidance on tool development, improvement, evaluation, and selection for developers, researchers, and potential users.
@inproceedings{LiCFFLLLC23,
title = {Comparison and Evaluation on Static Application Security Testing (SAST) Tools for Java},
author = {Kaixuan Li and Sen Chen and Lingling Fan and Ruitao Feng and Han Liu and Chengwei Liu and Yang Liu and Yixiang Chen},
booktitle = {FSE},
year = {2023},
doi = {10.1145/3611643.3616262}
}