一句话：首次系统评估 LLM 做配置验证的可行性与有效性，提出基于 few-shot 提示与输出检查的通用框架 Ciri。
配置错误是软件故障的主要成因，而现有实践依赖开发者手写规则或测试用例来验证配置值，代价高昂；基于机器学习的方法则受制于大规模现场数据需求与系统专用模型。本文首次系统分析用大语言模型（LLM）做配置验证的可行性与有效性，提出通用验证框架 Ciri：它结合合法配置与错误配置数据进行 few-shot 提示工程，并对 LLM 输出加以检查，以缓解幻觉与不确定性。作者在 8 个流行 LLM 与 10 个广泛部署的开源系统的配置数据上评估验证效果，结果（1）确认了 LLM 用于配置验证的潜力，（2）探索了 LLM 验证器的设计空间，（3）揭示了诸如对某些类型错误配置检测无效、对常见配置参数存在偏向等开放挑战。
Misconfigurations are major causes of software failures. Existing practices rely on developer-written rules or test cases to validate configuration values, which are expensive. Machine learning (ML) for configuration validation is considered a promising direction, but has been facing challenges such as the need of large-scale field data and system-specific models. Recent advances in Large Language Models (LLMs) show promise in addressing some of the long-lasting limitations of ML-based configuration validation. We present the first analysis on the feasibility and effectiveness of using LLMs for configuration validation. We empirically evaluate LLMs as configuration validators by developing a generic LLM-based configuration validation framework, named Ciri. Ciri employs effective prompt engineering with few-shot learning based on both valid configuration and misconfiguration data. Ciri checks outputs from LLMs when producing results, addressing hallucination and nondeterminism of LLMs. We evaluate Ciri's validation effectiveness on eight popular LLMs using configuration data of ten widely deployed open-source systems. Our analysis (1) confirms the potential of using LLMs for configuration validation, (2) explores design space of LLM-based validators like Ciri, and (3) reveals open challenges such as ineffectiveness in detecting certain types of misconfigurations and biases towards popular configuration parameters.
DOI 原文 ·
@inproceedings{paperbot8711,
title = {Large Language Models as Configuration Validators},
author = {Xinyu Lian and Yinfang Chen and Runxiang Cheng and Jie Huang and Parth Thakkar and Minjia Zhang and Tianyin Xu},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00017}
}