一句话：提出用RAG验证配置依赖，并发现需按失败分析定制上下文才能显著提升LLM验证效果。
配置依赖指多技术需协调设置才能正确交互；现有检测误报高、依赖额外验证且局限于特定项目/技术，LLM验证也受项目/技术差异与上下文缺失影响。本文提出用检索增强生成（RAG）进行配置依赖验证，引入项目与技术特定上下文，并通过大规模实证研究评估RAG能否改善LLM验证、需要何种上下文。结果显示，原始LLM已有较强验证能力，朴素RAG效果边际甚至负面；依据验证失败定性分析定制上下文后，所有模型显著提升，平均精确率0.84、召回率0.70，较原始LLM分别提升35%和133%，并揭示非定制信息未必有益。
Configuration dependencies arise when multiple technologies in a software system require coordinated settings for correct interplay. Existing approaches for detecting such dependencies often yield high false-positive rates, require additional validation mechanisms, and are typically limited to specific projects or technologies. Recent work that incorporates large language models (LLMs) for dependency validation still suffers from inaccuracies due to project- and technology-specific variations, as well as from missing contextual information.In this work, we propose to use retrieval-augmented generation (RAG) systems for configuration dependency validation, which allows to incorporate additional project- and technology-specific context information. Specifically, we evaluate whether RAG can improve LLM-based validation of configuration dependencies and what contextual information are needed to overcome the static knowledge base of LLMs. To this end, we conducted a large empirical study on validating configuration dependencies using RAG. Our evaluation shows that vanilla LLMs already demonstrate solid validation abilities, while RAG has only marginal or even negative effects on the validation performance of the models. By incorporating tailored contextual information into the RAG system–derived from a qualitative analysis of validation failures–we achieve significantly more accurate validation results across all models, with an average precision of 0.84 and recall of 0.70, representing improvements of 35% and 133% over vanilla LLMs, respectively. In addition, these results offer two important insights: Simplistic RAG systems may not benefit from additional information if it is not tailored to the task at hand, and it is often unclear upfront what kind of information yields improved performance.
DOI 原文 ·
@inproceedings{paperbot9284,
title = {On Automating Configuration Dependency Validation via Retrieval-Augmented Generation},
author = {Sebastian Simon and Alina Mailach and Johannes Dorn and Norbert Siegmund},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00207}
}