一句话：提出变异式模糊测试框架 Quirx，系统评估 LLM 提示鲁棒性并揭示不同任务与模型间的失效模式。
大语言模型日益支撑关键业务流程，但提示鲁棒性研究不足：同义词替换或指令重排等微小变化即可导致输出显著偏移，而现有评估多依赖手工测试，难以规模化。本文提出 Quirx，一个基于变异的模糊测试框架，对提示施加词汇、语义、结构三个维度的变异，在目标模型上执行，并通过多级相似度分析度量响应一致性，输出鲁棒性分数、失效模式并支持模型选型。作者在 GPT-3.5-turbo、GPT-4o-mini、Claude-3.5-Sonnet 与 Claude-Sonnet-4 四个模型、三类任务上评估：情感分类鲁棒性一致为 1.00；摘要任务对提供方高度敏感（0.23–0.58），Claude 模型鲁棒性约为 OpenAI 的 2.5 倍；SQL 生成表现稳定（0.80–1.00）。结构变异造成摘要任务 50–67% 的失败，但对其他任务影响很小。
Large Language Models (LLMs) increasingly power critical business processes, yet prompt robustness remains under-explored. Small variations—such as synonym changes or instruction reordering—can cause significant output shifts, undermining reliability in domains like customer service and finance. Existing evaluations rely on ad-hoc manual testing, limiting scalability in production environments.We present Quirx, a mutation-based fuzzing framework for systematically evaluating prompt robustness across LLM providers. Quirx applies tri-dimensional mutations (lexical, semantic, structural), executes them against target models, and measures response consistency via multi-level similarity analysis. It produces robustness scores, reveals failure patterns, and supports informed model selection.We evaluate Quirx on four models (GPT-3.5-turbo, GPT-4o-mini, Claude-3.5-Sonnet, Claude-Sonnet-4) across three tasks. Results show sentiment classification is uniformly robust (1.00), summarization is highly provider-sensitive (0.23–0.58) with Claude models 2.5× more robust than OpenAI, and SQL generation is consistently strong (0.80–1.00). Structural mutations cause 50–67% of summarization failures but have minimal effect on other tasks.Demo video: https://youtu.be/Sm3Gk2X2-vk
DOI 原文 ·
@inproceedings{paperbot9225,
title = {Quirx: A Mutation-Based Framework for Evaluating Prompt Robustness in LLM-based Software},
author = {Souhaila Serbout},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00383}
}