一句话：提出 CAFFE，一个意图感知的反事实公平性测试框架，用结构化测试用例与语义相似度更可靠检测 LLM 偏见。
大语言模型日益成为软件系统的基础组件，其公平性问题备受关注。现有工作多借助蜕变测试，通过输入变换发现模型行为不一致。本文提出 CAFFE——反事实公平性评估框架，将 LLM 公平性测试用例形式化为提示意图、对话上下文、输入变体、期望公平阈值和测试环境配置等组件；框架可自动生成针对性测试数据，并用语义相似度指标评估模型响应。作者在三个不同架构家族的大语言模型上开展实验，结果表明 CAFFE 比现有蜕变方法具有更广的偏见覆盖和更可靠的公平性违规检测。
Nowadays, Large Language Models (LLMs) are foundational components of modern software systems. As their influence grows, concerns about fairness have become increasingly pressing. Prior work has proposed metamorphic testing to detect fairness issues, applying input transformations to uncover inconsistencies in model behavior. This paper introduces an alternative perspective for testing counterfactual fairness in LLMs, proposing a structured and intent-aware framework coined CAFFE (Counterfactual Assessment Framework for Fairness Evaluation). Inspired by traditional non-functional testing, CAFFE (1) formalizes LLM-Fairness test cases through explicitly defined components, including prompt intent, conversational context, input variants, expected fairness thresholds, and test environment configuration, (2) assists testers by automatically generating targeted test data, and (3) evaluates model responses using semantic similarity metrics. Our experiments, conducted on three different architectural families of LLM, demonstrate that CAFFE achieves broader bias coverage and more reliable detection of unfair behavior than existing metamorphic approaches.
@inproceedings{abs-2512-16816,
title = {Toward Systematic Counterfactual Fairness Evaluation of Large Language Models: The CAFFE Framework},
author = {Alessandra Parziale and Gianmario Voria and Valeria Pontillo and Gemma Catolino and Andrea De Lucia and Fabio Palomba},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3787848}
}