一句话：提出VADA框架与多文化价值基准，用贝叶斯集成评估器自动测试LLM的文化价值对齐程度。
随着LLM进入教育、法律咨询等全球部署的软件系统，其行为是否符合多元文化价值成为软件工程难题：单元测试与形式验证擅长功能需求，却难以编码和验证文化敏感性、道德适当性等规范性期望。为此，作者提出VADA——一个在多文化价值体系下测试与基准评测LLM的框架。VADA通过模块化流水线生成基于文化价值维度的场景与问题，覆盖中国核心价值观、欧盟基本权利框架和伊斯兰伦理框架中的25个价值维度。评估端采用贝叶斯集成，根据每个评估器观测到的可靠性分配维度特定权重，并利用VADA生成的数据微调轻量级监督评估器，提供快速可扩展的替代方案。该流水线构建了包含11,865个自动标注案例的大型基准，以及1,000个人工标注子集用于验证。VADA显著优于基于提示的评估器，达到93.6%准确率，与人工标注高度一致。消融实验表明评估器多样性与可靠性加权均有贡献。VADA还支持对近期LLM进行对比审计，揭示其在各文化维度上对齐不一致。
Large language models (LLMs) now sit inside an expanding set of intelligent software systems, from education assistants to legal advisory tools. When these systems ship to global users, ensuring their behavior respects diverse cultural values has become a pressing challenge for software engineers. Traditional software engineering methods such as unit testing and formal verification work well for functional requirements, but they struggle to encode and validate normative expectations like cultural sensitivity or moral appropriateness. To address this gap, we introduce VADA, a framework for testing and benchmarking LLMs under multiple cultural value systems. VADA builds test cases through a modular pipeline that generates scenarios and questions grounded in culturally defined value dimensions. The resulting suite covers 25 value dimensions drawn from China’s Core Values, EU’s fundamental rights framework, and Islamic ethical frameworks. VADA uses a Bayesian ensemble and assigns each evaluator a dimension specific weight based on its observed reliability to evaluate model responses, and we additionally fine tune a lightweight supervised evaluator using VADA generated data, offering a fast and scalable alternative to running multiple large evaluators. Using this pipeline, we build a large benchmark containing 11,865 automatically annotated cases, along with a human labeled subset of 1,000 instances for validation and evaluation. VADA substantially outperforms current prompt-based evaluators, reaching 93.6% accuracy and showing strong alignment with human annotations. Ablation studies indicate that both evaluator diversity and reliability based weighting contribute to performance. VADA also supports comparative audits of recent LLMs and reveals inconsistent alignment across cultural dimensions. Together, these results suggest that VADA provides a robust basis for value aware evaluation and supports the development of culturally aligned LLM based systems.
@inproceedings{paperbot11884,
title = {VADA: A Multicultural Benchmark for Value-Aware Data Generation and Alignment Evaluation in LLMs},
author = {Zhenlun Zhang and Yang Feng and Shihao Weng and Yining Yin and Jincheng Li and Jia Liu},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3787807}
}