一句话：提出 REAG 与置信度校准级联判定，为 LLM 软件自动生成上下文相关的测试预言并给出可靠判定。
基于 LLM 的软件（LBS）把大模型作为核心组件，其输出随用户画像与软件上下文而变且具有随机性，使经典验收测试与测试预言失效。本文提出一个具备判定可靠性校准的自动化验收测试框架，包含两项贡献：其一，需求增强生成 REAG，通过自适应 RAG 检索软件需求、领域知识与用户画像，并结合自推理生成上下文相关的测试预言；其二，针对检索到无关约束、意图误解或需求幻觉，提出置信度校准的级联判定，以模拟专家一致性量化判定可靠性，接受高置信结论、升级模糊案例或在不确定时弃权，并由 conformal risk control 提供经验可靠性保证。在生产级营养咨询应用上的工业案例研究表明，REAG 的预言质量达 3.91/5，82% 的案例达到合格或边缘质量；级联判定准确率 98.8%，经过滤不合格输出把预言质量提升至 4.30，且相比单一判定基线成本效率提升 31.7%，验证了工业可行性。
LLM-based software (LBS) integrates large language models as core components to deliver flexible and personalised responses. Unlike traditional software with deterministic outputs, LBSs exhibit context-dependent, stochastic behaviour that renders classical acceptance testing and test oracles insufficient: the same query may require fundamentally different responses depending on users' personas and software context. This gap creates an urgent need for automated acceptance testing frameworks that can autonomously interpret users' instructions, while also raising the challenge of how we can reliably interpret users' intentions in a changing environment. In this paper, we present an automated acceptance testing framework for LBS with calibrated verdict reliability through two technical contributions. First, we introduce REquirements-Augmented Generation (REAG), which interprets user intentions by retrieving relevant software requirements, domain knowledge, and personas via adaptive RAG and self-reasoning to generate context-aware test oracles. Second, recognising that oracle generation may retrieve irrelevant constraints, misinterpret intent, or hallucinate requirements, we introduce a confidence-calibrated cascade judgment that quantifies verdict reliability via simulated expert agreement, accepting high-confidence verdicts, escalating ambiguous cases, or abstaining when uncertain, with empirical reliability guarantees backed by conformal risk control. An industrial case study on a production nutrition advisory application demonstrates that REAG achieves a 3.91/5 oracle quality score, successfully achieving qualified or marginal oracle quality in 82% of cases. The confidence-calibrated cascade achieves 98.8% accuracy, improves oracle quality from 3.91 to 4.30 by filtering unqualified outputs, and delivers a 31.7% cost-efficiency improvement over single-judge baselines, validating industrial viability.
@inproceedings{paperbot12449,
title = {Requirements-Augmented Generation for Trustworthy Acceptance Testing of LLM-Based Software},
author = {Fanyu Wang and Chetan Arora and Zhenping Xie and Yonghui Liu and Kla Tantithamthavorn and Aldeida Aleti and Siwei Jiang},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834491}
}