一句话：提出 PenForge，在渗透测试中动态构建 LLM 专家代理，零日设定下漏洞利用成功率提升 3 倍。
论文关注 LLM 驱动的自动化渗透测试：现有方法或依赖单一通用代理而难以处理复杂场景，或依赖预先准备的狭窄专用代理而无法适应多样漏洞。PenForge 提出在测试过程中动态构建专家代理，将潜在攻击面的自动侦察与按需实例化的上下文感知代理相结合，从而实现针对性漏洞利用。在 CVE-Bench 的零日设定（尤其困难）上，PenForge 达到 30.0% 的漏洞利用成功率（12/40），相比当前最优方法提升 3 倍。论文还指出未来方向：提供更丰富的工具使用知识、扩展包含更多漏洞与攻击类型的基准、以及通过可解释机制与人工审查提升开发者信任。
Penetration testing is essential for identifying vulnerabilities in web applications before real adversaries can exploit them. Recent work has explored automating this process with Large Language Model (LLM)-powered agents, but existing approaches either rely on a single generic agent that struggles in complex scenarios or narrowly specialized agents that cannot adapt to diverse vulnerability types. We therefore introduce PenForge, a framework that dynamically constructs expert agents during testing rather than relying on those prepared beforehand. By integrating automated reconnaissance of potential attack surfaces with agents instantiated on the fly for context-aware exploitation, PenForge achieves a 30.0% exploit success rate (12/40) on CVE-Bench in the particularly challenging zero-day setting, which is a 3 × improvement over the state-of-the-art. Our analysis also identifies three opportunities for future work: (1) supplying richer tool-usage knowledge to improve exploitation effectiveness; (2) extending benchmarks to include more vulnerabilities and attack types; and (3) fostering developer trust by incorporating explainable mechanisms and human review. As an emerging result with substantial potential impact, PenForge embodies the early-stage yet paradigm-shifting idea of on-the-fly agent construction, marking its promise as a step toward scalable and effective LLM-driven penetration testing.
@inproceedings{paperbot12072,
title = {PenForge: On-the-Fly Expert Agent Construction for Automated Penetration Testing},
author = {Huihui Huang and Jieke Shi and Junkai Chen and Ting Zhang and Yikun Li and Chengran Yang and Eng Lieh Ouh and Lwin Khin Shar and David Lo},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3786582.3786814}
}