一句话：提出 LLMCapLens 动态基准生成框架，系统测量并增强 LLM 的静态污点分析能力。
LLM 被越来越多地用于静态污点分析，但学界缺乏对其能力的系统理解；现有基准存在训练数据泄露、未考虑 LLM 特性与评估标准不当等问题，人工构建则成本高且难随模型演进。为此，本文提出 LLMCapLens 动态基准生成框架，建模影响因子，用基于基本单元的生成与轻量级动态污点分析验证实现自动化、多样化且正确的基准生成，并提出测量驱动、免训练、模型特定的增强方法。在 10 个主流 LLM 上的评测揭示了各因子下的表现与模型特有错误成因；增强使 GPT-4 Turbo 在 19 个因子中的 16 个上提升，真负率平均提高 21.29%，并通过漏洞检测验证了实际效果。
LLMs offer a promising avenue to overcome the limitations of traditional taint analysis techniques, with a growing number of studies leveraging LLMs for taint analysis and its downstream applications. However, these studies lack a systematic understanding of LLMs’ taint analysis capabilities, limiting their transferability and reliability. To bridge this gap and better apply LLMs to static taint analysis, we aim to comprehensively measure and understand LLMs’ taint analysis capabilities.Using existing benchmarks is a straightforward approach, but they are unsuitable due to issues such as training data leakage, not accounting for LLMs’ features, and improper assessment criteria. Manually constructing new benchmarks is not only labor-intensive but also struggles to remain effective as LLMs evolve. To address these, we propose LLMCapLens, a dynamic benchmark generation framework to systematically measure and enhance LLMs’ capabilities. LLMCapLens models influencing factors of LLMs’ taint analysis capabilities, employing a Basic Unit-Based generation method and a lightweight dynamic taint analysis-based verification method to implement the automated generation of targeted benchmarks, ensuring both diversity and correctness. Furthermore, LLMCapLens proposes a measurement-driven, training-free, model-specific enhancement approach.We apply LLMCapLens to 10 mainstream LLMs, revealing how they perform under various influencing factors and identifying unique characteristics, such as the underlying error causes for each model. Notably, our enhancement approach significantly improves LLM performance—GPT-4 Turbo, for instance, achieved improvements across 16 out of 19 factors, with an average True Negative Rate increase of 21.29%. Finally, we validate the real-world impact of our method by applying enhanced LLMs to vulnerability detection, demonstrating a substantial improvement over prior approaches.
DOI 原文 ·
@inproceedings{paperbot9355,
title = {Exploring Static Taint Analysis in LLMs: A Dynamic Benchmarking Framework for Measurement and Enhancement},
author = {Haoran Zhao and Lei Zhang and Keke Lian and Fute Sun and Bofei Chen and Yongheng Liu and Zhiyu Wu and Yuan Zhang and Min Yang},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00082}
}