一句话：提出首个评估 LLM 漏洞检测推理轨迹忠实性的框架 VulFaith，揭示不忠实普遍存在且可通过数据增强缓解。
LLM 生成的推理轨迹通过暴露据称支撑其预测的因素来提供可解释性，但其有用性取决于忠实性，即是否反映模型实际依赖的因素；不忠实的轨迹可能只是与真实决策过程不符的合理化叙述。然而，基于 LLM 的漏洞检测中推理轨迹忠实性尚未被系统研究。本文提出 VulFaith，首个该领域的推理轨迹忠实性评估框架，形式化两条互补准则：完备性要求所有支撑不安全预测的决策相关因素都被暴露；逐因素忠实性要求每个被引用因素确实被模型依赖。为落实准则，作者设计基于自动修复的评估方案，通过代码修复移除被引用的不安全因素并重新查询模型，检验预测是否仍然维持。作者在仓库级与函数级漏洞数据集上评估了十个先进开源模型与四个商业 LLM。结果显示不忠实普遍存在，完备性不忠实最高达 66.14%，逐因素不忠实最高达 94.00%；进一步验证所识别的不忠实案例可用于监督微调的数据增强，使完备性不忠实最多降低 47.89%，逐因素不忠实最多降低 59.18%。
Reasoning traces generated by large language models (LLMs) provide interpretability by exposing the factors that purportedly support their predictions. Such interpretability is important because model decisions may rely on spurious factors, and it can help practitioners identify such reliance and respond appropriately. However, the usefulness of reasoning traces hinges on their faithfulness, namely whether they reflect the factors that the model actually relies on when producing its prediction. Unfaithful traces may instead serve as plausible narratives that do not correspond to the true decision process. Despite its importance, reasoning-trace faithfulness has not been systematically studied in LLM-based vulnerability detection. We present VulFaith, the first framework for evaluating reasoning-trace faithfulness in this domain. We formalize two complementary criteria: completeness, which requires that all decision-relevant factors sustaining an insecure prediction are exposed, and per-factor faithfulness, which requires that each cited factor is genuinely relied upon by the model. To operationalize these criteria, we design an automated repair-based evaluation scheme that removes cited insecurity factors through code repairs and re-queries the model to test whether the prediction remains sustained. We implement VulFaith and conduct experiments on both repository-level and function-level vulnerability datasets, evaluating ten state-of-the-art open-source models and four commercial LLMs. Our results show that unfaithfulness is widespread, with completeness unfaithfulness reaching up to 66.14% and per-factor unfaithfulness up to 94.00%. We further validate that the unfaithfulness cases identified by VulFaith can support data augmentation for supervised fine-tuning, reducing completeness unfaithfulness by up to 47.89% and per-factor unfaithfulness by up to 59.18%.
@inproceedings{paperbot12391,
title = {Are Reasoning Traces Faithful in LLM-Based Code Vulnerability Detection?},
author = {Jialai Wang and Juncheng Lu and Songtao Yang and Bingyi He and Zeyu Gao and Chao Zhang and Shuai Wang},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834364}
}