一句话：提出评估模型并对 6 个 Java SCA 工具进行大规模实证，揭示依赖解析与漏洞检测的精度问题。
SCA 工具用于检测第三方开源库引入的漏洞，但面对构件格式、依赖导入与依赖声明多样化，Java 场景仍缺乏全面评估。本文提出含扫描模式、扫描方法和 Maven SCA 范围（SSM）的评估模型，先定性考察 6 个 SCA 工具，再在 21,130 个 Maven 模块、73,499 个唯一依赖的数据集上，于构建扫描与预构建扫描下定量评估依赖和漏洞精度。结果显示多数工具未完全支持 SSM，导致精度受损：依赖检测平均 F1 为 0.890 与 0.692，漏洞检测平均 F1 仅 0.475；良好支持 SSM 可降低依赖误报 34.24%、漏报 6.91%，以及漏洞误报 18.28%、漏报 8.72%。
Software composition analysis (SCA) tools are proposed to detect potential vulnerabilities introduced by open-source software (OSS) imported as third-party libraries (TPL). With the increasing complexity of software functionality, SCA tools may encounter various scenarios during the dependency resolution process, such as diverse formats of artifacts, diverse dependency imports, and diverse dependency specifications. However, there still lacks a comprehensive evaluation of SCA tools for Java that takes into account the above scenarios. This could lead to a confined interpretation of comparisons, improper use of tools, and hinder further improvements of the tools. To fill this gap, we proposed an Evaluation Model which consists of Scan Modes, Scan Methods, and SCA Scope for Maven (SSM), for comprehensive assessments of the dependency resolving capabilities and effectiveness of SCA tools. Based on the Evaluation Model, we first qualitatively examined 6 SCA tools’ capabilities. Next, the accuracy of dependency and vulnerability is quantitatively evaluated with a large-scale dataset (21,130 Maven modules with 73,499 unique dependencies) under two Scan Modes (i.e., build scan and pre-build scan). The results show that most tools do not fully support SSM, which leads to compromised accuracy. For dependency detection, the average F1-score is 0.890 and 0.692 for build and pre-build respectively, and for vulnerability accuracy, the average F1-score is 0.475. However, proper support for SSM reduces dependency detection false positives by 34.24% and false negatives by 6.91%. This further leads to a reduction of 18.28% in false positives and 8.72% in false negatives in vulnerability reports.
@inproceedings{ZhaoCXLZWSL23,
title = {Software Composition Analysis for Vulnerability Detection: An Empirical Study on Java Projects},
author = {Lida Zhao and Sen Chen and Zhengzi Xu and Chengwei Liu and Lyuye Zhang and Jiahui Wu and Jun Sun and Yang Liu},
booktitle = {FSE},
year = {2023},
doi = {10.1145/3611643.3616299}
}