一句话：提出格式感知蜕变测试框架，系统揭示 LLM 文档工作流因文档格式变化而出现的准确率下降与决策漂移。
论文关注 LLM 驱动软件系统转向以文档为中心的端到端工作流后，同一语义内容以不同文档格式（如 CSV）上传时是否仍保持稳健。现有测试多聚焦单一提示字符串输入，忽略格式因素。作者提出格式感知蜕变测试框架，定义三条蜕变关系，并在四个代表性 LLM 工作流、四类真实任务和四种文档格式上执行 48,000 次运行。结果显示格式变化构成系统性威胁：仅切换格式即可使准确率最高下降 53.63%，超过 41% 的实例发生决策漂移。作者进一步设计面向用户的轻量缓解策略，无需重训练即可恢复最高 44.21% 的格式诱导决策漂移，并指出文档格式并非中性封装，而是影响 LLM 软件可靠性的关键因素。
LLM-driven software systems are rapidly evolving from plain-text conversations to document-centric end-to-end workflows, where the same semantic content can be delivered in diverse document formats (e.g., CSV) through file upload interfaces. Yet existing testing work focuses on the robustness and reliability of models and systems whose input is a single prompt string, leaving a critical question unanswered: Can these document workflows maintain robust behaviors when the same content arrives in a different document format? To fill the gap, in this paper, we propose a format-aware metamorphic testing framework with three metamorphic relations to comprehensively evaluate the format robustness of end-to-end LLM document workflows. Based on this framework, we conduct a large-scale empirical study spanning four representative LLM workflows, four real-world tasks, and four document formats, comprising a total of 48,000 workflow executions. Our findings reveal that format variation poses a systematic and serious threat. Merely switching formats can cause accuracy to drop by up to 53.63% and trigger decision drifts in over 41% of instances. We further design lightweight mitigation strategies from the users’ perspective that recover up to 44.21% of format-induced decision drift without model retraining. Our study demonstrates that document format is not a neutral wrapper but a critical factor affecting the reliability of LLM software systems, calling for corresponding testing and safeguards in the deployment in real-world high-stakes scenarios.
@inproceedings{abs-2607-27648,
title = {Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows},
author = {Xiaoyu Zhang and Xianyun Cheng and Tianlin Li and Yuwei Zheng and Yue Yang and Yang Liu},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3837456}
}