一句话：在 PrimeVul Pair 上分阶段实证：推理模式主要靠召回提升 F1，双智能体设计 F1 最高但告警负担沉重，解释质量提升有限。
大语言模型被越来越多地用于软件漏洞检测，但新近的“思考”（reasoning）模式带来的收益仍不明确；同时，智能体式设计希望借结构化协作增强分析，而实际安全流程还需要可检验、可验证、可修复的解释。已有研究往往把这些维度分开考察或设置差异很大，难以厘清各因素各自的贡献。本文在 PrimeVul Pair 基准上开展分阶段、因子分离的实证研究：第一阶段在无智能体条件下，以零样本与少样本提示比较四个开源模型的 instruct 与 thinking 变体；第二阶段评估单智能体、双智能体与多智能体（SA/DA/MA）设计，并加入调用次数匹配的单智能体基线、告警负担指标与成本分析；第三阶段通过盲测人工评估与校准后的 LLM 评估，在控制回答长度与模型家族的前提下，对正确与错误预测的解释质量进行考察。结果显示：thinking 在 8 个设置中的 7 个提升了非智能体 F1，主要来自召回率；DA 取得最高 F1/召回，但会将 88.2% 的函数标记为脆弱；MA 在成对判别上更好，而调用匹配的自我修订（self-revision）总体 P-C 最优。thinking 能耗高 2.7–4.4 倍，且其 F1 收益随编排复杂度上升而下降；解释质量的提升因模型而异、部分由长度中介，并且无法区分正确与错误预测。
Large language models (LLMs) are increasingly studied for software vulnerability detection, but the benefits of recent reasoning-enabled (“thinking”) modes remain unclear. At the same time, agentic LLM designs aim to enhance analysis through structured collaboration, while practical software security workflows require explanations that facilitate inspection, validation, and remediation. Because prior studies often examine these dimensions separately or under heterogeneous settings, their individual effects on vulnerability detection remain difficult to disentangle. This paper addresses the gap through a phased, factor-separated empirical study on the PrimeVul Pair benchmark. In the first phase, we compare instruct and thinking variants of four open models under zero- and few-shot prompting without agents. In the second phase, we evaluate single-, dual-, and multi-agent (SA, DA, and MA) designs alongside call-matched single-agent baselines, alert-burden metrics, and cost analysis. In the third, we assess explanations through blinded human and calibrated LLM evaluation across correct and incorrect predictions, controlling for response length and model family. Thinking improves non-agent (NA) F1 in 7 of 8 settings, mainly through recall, while DA achieves the highest F1/recall but flags 88.2% of functions as vulnerable. MA performs better on pairwise discrimination, although call-matched self-revision achieves the best Pair-wise Correct Prediction (P-C) overall. Thinking also uses 2.7–4.4× more energy, and its F1 benefit declines with greater orchestration. Explanation gains are model-specific, partly length-mediated, and do not distinguish correct from incorrect predictions.
@inproceedings{paperbot12446,
title = {To Think or Not to Think: Evaluating LLM Reasoning and Agents in Vulnerability Detection},
author = {Hua Beng Tan and Ratnadira Widyasari and Merve Astekin and Arda Goknil and Hasan Sözer and Yan Naing Tun and Erik Johannes Husom and Lwin Khin Shar and Lingxiao Jiang},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3837521}
}