一句话：提出首个演化式测试框架 ORFuzz，系统检测并分析 LLM 的过度拒绝行为，并构建高可迁移的测试基准 ORFuzzSet。
大语言模型因安全策略过于保守而错误拒绝良性查询，即"过度拒绝"，这一功能缺陷严重损害其可靠性与可用性；作者通过用户研究指出，现有测试方法存在基准缺陷、用例生成能力不足等问题。本文提出首个面向 LLM 过度拒绝的系统化演化测试框架 ORFuzz，集成三个核心组件：安全类别感知的种子选择以覆盖各类安全场景；利用推理型 LLM 自适应优化变异算子以生成有效用例；以及与人类判断对齐、经校验能准确反映用户对有害性与拒绝感知的判定模型 OR-Judge。评估表明，ORFuzz 生成多样化且经验证的过度拒绝实例的平均比率达 6.98%，是领先基线的两倍以上；其产出进一步构成 ORFuzzSet 基准，含 1786 条高可迁移测试用例，在 14 个不同 LLM 上取得 57.37% 的平均过度拒绝率，显著优于现有数据集。
Large Language Models (LLMs) have been found to show over-refusal problems—erroneously rejecting benign queries due to overly conservative safety measures—a critical functional flaw that undermines their reliability and usability. Current methods for testing this behavior are demonstrably inadequate, suffering from flawed benchmarks and limited test generation capabilities, as highlighted by our empirical user study. To the best of our knowledge, this paper introduces the first evolutionary testing framework, ORFuzz, for the systematic detection and analysis of LLM over-refusals. ORFuzz uniquely integrates three core components: (1) safety category-aware seed selection for comprehensive test coverage, (2) adaptive mutator optimization using reasoning LLMs to generate effective test cases, and (3) OR-Judge, a human-aligned judge model validated to accurately reflect user perception of toxicity and refusal. Our extensive evaluations demonstrate that ORFuzz generates diverse, validated over-refusal instances at a rate (6.98% average) more than double that of leading baselines, effectively uncovering vulnerabilities. Furthermore, ORFuzz’s outputs form the basis of ORFuzzSet, a new benchmark of 1,786 highly transferable test cases that achieves a superior 57.37% average over-refusal rate across 14 diverse LLMs, significantly outperforming existing datasets. ORFuzz and ORFuzzSet provide a robust automated testing framework and a valuable community resource, paving the way for developing more reliable and trustworthy LLM-based software systems. The code of this paper is available at: https://github.com/HotBento/ORFuzz.
@inproceedings{paperbot9229,
title = {ORFuzz: Fuzzing the "Other Side" of LLM Safety - Testing Over-Refusal},
author = {Haonan Zhang and Dongxia Wang and Yi Liu and Kexin Chen and Jiashui Wang and Xinlei Ying and Long Liu and Wenhai Wang},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00156}
}