一句话：首次系统研究注意力神经网络真实故障，提出七类注意力特有故障分类和四条诊断启发式。
现有深度学习故障分类难以覆盖注意力机制引入的独特故障，使实践者缺少可操作诊断指导。本文对来自 GitHub、Hugging Face、Stack Overflow 等十种框架、96 个项目的 555 个真实故障进行首次系统性实证研究，构建包含七类注意力特有故障的新分类体系。结果显示，超过半数 ABNN 故障源于注意力架构特有机制。作者进一步分析根因、表现形式与症状，并通过症状—根因关联提炼出四条基于证据的诊断启发式，可解释 33.0% 的注意力特有故障，为 ABNN 故障诊断提供系统化、分类驱动的指导。
Attention mechanisms are at the core of modern neural architectures, powering systems ranging from ChatGPT to autonomous vehicles, and driving a major economic impact. However, high-profile failures, such as ChatGPT’s nonsensical outputs or Google’s suspension of Gemini’s image generation due to attention weight errors, highlight a critical gap: existing deep learning fault taxonomies might not adequately capture the unique failures introduced by attention mechanisms. This gap leaves practitioners without actionable diagnostic guidance. To address this gap, we present the first comprehensive empirical study of faults in attention-based neural networks (ABNNs). Our work is based on a systematic analysis of 555 real-world faults collected from 96 projects across ten frameworks, including GitHub, Hugging Face, and Stack Overflow. Through our analysis, we develop a novel taxonomy comprising seven attention-specific fault categories, not captured by existing work. Our results show that over half of the ABNN faults arise from mechanisms unique to attention architectures. We further analyze the root causes and manifestations of these faults through various symptoms. Finally, by analyzing symptom–root cause associations, we identify four evidence-based diagnostic heuristics that explain 33.0% of attention-specific faults and provide systematic, taxonomy-driven guidance for diagnosing faults in ABNNs.
@inproceedings{abs-2508-04925,
title = {Why Attention Fails: A Taxonomy of Faults in Attention-Based Neural Networks},
author = {Sigma Jahan and Saurabhsingh Rajput and Tushar Sharma and Masud Rahman},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3773118}
}