一句话：首个针对专业开发者在真实项目中使用的IDE集成AI漏洞检测与修复工具的用户研究，发现其因高误报率与不可用修复而不适合实际应用。
安全漏洞给用户和组织带来巨大成本，尽早检测与修复至关重要。已有研究表明深度学习模型能有效检测漏洞，但从基准测试到实际应用的迁移研究甚少。本文首次对专业软件开发者在其自有真实项目中使用漏洞检测与修复工具进行实证研究。作者实现了DeepVulguard——一个基于最新检测与修复模型的IDE集成工具，能扫描漏洞（识别漏洞类型与脆弱代码区域）、建议修复、通过聊天界面提供自然语言解释。研究招募了17名微软专业开发者，观察其在自有代码上使用该工具，并访谈评估有用性、速度、信任度、相关性与工作流整合。参与者共扫描24个项目、6.9k文件、超170万行源代码，生成170条告警与50条修复建议。结果显示，尽管最先进的AI检测与修复工具展现潜力，但因高误报率和不可应用的修复，尚不实用。用户反馈揭示了从上下文不完整到缺乏代码库定制等一系列可操作的痛点，并探讨了置信度分数、解释与聊天交互等AI功能的作用。
Security vulnerabilities impose significant costs on users and organizations. Detecting and addressing these vulnerabilities early is crucial to avoid exploits and reduce development costs. Recent studies have shown that deep learning models can effectively detect security vulnerabilities. Yet, little research explores how to adapt these models from benchmark tests to practical applications, and whether they can be useful in practice. This paper presents the first empirical study of a vulnerability detection and fix tool with professional software developers on real projects that they own. We implemented DeepVulguard, an IDE-integrated tool based on state-of-the-art detection and fix models, and show that it has promising performance on benchmarks of historic vulnerability data. DeepVulguard scans code for vulnerabilities (including identifying the vulnerability type and vulnerable region of code), suggests fixes, provides natural-language explanations for alerts and fixes, leveraging chat interfaces. We recruited 17 professional software developers at Microsoft, observed their usage of the tool on their code, and conducted interviews to assess the tool's usefulness, speed, trust, relevance, and workflow integration. We also gathered detailed qualitative feedback on users' perceptions and their desired features. Study participants scanned a total of 24 projects, 6.9 k files, and over 1.7 million lines of source code, and generated 170 alerts and 50 fix suggestions. We find that although state-of-the-art AI-powered detection and fix tools show promise, they are not yet practical for real-world use due to a high rate of false positives and non-applicable fixes. User feedback reveals several actionable pain points, ranging from incomplete context to lack of customization for the user's codebase. Additionally, we explore how AI features, including confidence scores, explanations, and chat interaction, can apply to vulnerability detection and fixing. Based on these insights, we offer practical recommendations for evaluating and deploying AI detection and fix models. Our code and data are available at this link: https://doi.org/10.6084/m9.figshare.26367139.
DOI 原文 ·
@inproceedings{paperbot8922,
title = {Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE},
author = {Benjamin Steenhoek and Kalpathy Sivaraman and Renata Saldivar Gonzalez and Yevhen Mohylevskyy and Roshanak Zilouchian Moghaddam and Wei Le},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00126}
}