一句话：通过202个PR实证揭示深度学习框架GPU优化的实际效果、22类策略及三类性能退化根因。
针对深度学习框架中大量GPU优化是否真正带来收益尚不清楚的问题，本文对主流框架的202个性能相关PR进行实证研究，归纳出22类优化策略，涵盖内核融合、自动调优、向量化等，跨越逻辑、执行与基础设施层。通过多层级基准测试和微架构剖析，作者发现优化收益并不稳定，部分策略显著加速，而相当比例收益有限甚至出现性能回退。研究进一步揭示三类反复出现的退化根因：激进融合导致寄存器压力上升、小内核中的同步开销迁移，以及高负载内核下的内存仲裁竞争。该工作首次系统刻画这些效应，为更可靠、性能感知的框架优化提供指导。
Performance optimization is a critical concern in the evolution of deep learning frameworks, as GPU execution dominates training and inference costs. While hundreds of optimizations are proposed to accelerate training and inference, it is unclear how often these changes deliver meaningful gains in practice, and why they sometimes fail. To bridge this gap, we conduct an empirical study of GPU performance optimizations in deep learning frameworks. We analyze 202 performance-related pull requests from the major deep learning frameworks, and categorize them into 22 distinct optimization strategies (e.g., kernel fusion, auto-tuning, and vectorization). This taxonomy reveals a rich set of developer-intended performance improvements spanning logic, execution, and infrastructure levels. Our evaluation reveals that optimization benefits are far from guaranteed. While some strategies achieve substantial speedups, a significant portion exhibit limited gains or even performance regressions when deployed in realistic workloads. To understand these discrepancies, we conduct multi-level benchmarking and microarchitectural profiling, uncovering three recurring root causes of performance degradation: (1) increased register pressure from aggressive fusion, (2) synchronization overhead migration in small kernels, and (3) memory arbitration under high-load kernels. These findings highlight a fundamental challenge in performance engineering: optimizations interact with hardware resources in complex and often counterintuitive ways, making their outcomes difficult to predict. Our study provides the first systematic characterization of these effects, offering insights to guide more reliable and performance-aware optimization in deep learning frameworks.
@inproceedings{paperbot12342,
title = {Bridging the Gap between Intent and Impact: An Empirical Study of GPU Optimizations in Deep Learning Frameworks},
author = {Siqi Wang and Fanjing Luo and Xing Hu and Xinyu Wang and Xin Xia},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834344}
}