一句话：刻画 GPU 归约与 GEMM 的位级决定因素，实现闭源库算术的黑盒重建、Triton 归约顺序强制与位等价静态验证。
浮点归约顺序、部分和精度、融合乘加与舍入位置使确定性 GPU 内核仍可能逐位不同，而被性能驱动的 tile shape 与自动调优难以判断配置是否位等价。本文系统刻画了归约与 GEMM 位级行为的决定因素，提出包含 split-K 中 K 划分的 GEMM 归约顺序描述符，并据此首次对闭源库（如 cuBLAS/rocBLAS）的算术进行黑盒重建：在 Blackwell 与 Hopper 上，其 Triton GEMM 家族在所有测试用例中与 NVIDIA cuBLAS 逐位一致，在带 fused epilogue 的真实 LLM 形状上性能持平或超过 torch.compile。其次，在 Triton lowering 中强制平衡树归约，并引入数据布局优化，使 GB300 与 H100 上 27 个内核中的 19 个进入自由顺序性能的 10% 以内。最后，为编译后 GPU 内核开发可靠的位等价静态检查器，首次覆盖 NVIDIA PTX 与 AMD GCN；集成进 Triton 自动调优器后可将搜索限制在单个位等价类内。
Determinism and numerical reproducibility are increasingly required of GPU kernels in machine learning systems, yet deterministic implementations of the same kernel can still differ bit for bit. Floating-point reduction order is the primary cause, alongside partial-sum precision, fused multiply-add operations, and rounding placement. These choices may be hand-coded, selected by a block-level language such as Triton, or hidden inside a closed-source library such as cuBLAS or rocBLAS. A tile shape chosen for performance therefore also determines the arithmetic, potentially breaking batch invariance. Preserving a fixed order can cost up to 20 percent, while an autotuner cannot identify which configurations are bitwise equivalent. We characterize the factors determining the bitwise behavior of reductions and general matrix multiplication (GEMM). First, we introduce a descriptor of GEMM reduction order, including the partitioning of K in split-K GEMM. Using it, we perform the first black-box reconstruction of a closed-source library's arithmetic for bit-level correctness. Our family of Triton GEMMs matches NVIDIA cuBLAS in all tested cases on Blackwell and Hopper. For realistic LLM shapes with fused epilogues, it matches or exceeds torch.compile performance. Second, we enforce balanced-tree reduction during Triton lowering and introduce a data-layout optimization that brings 19 of 27 kernels on GB300 and H100 within 10 percent of free-order performance. Third, we develop sound static checkers for bitwise equivalence between compiled GPU kernels, including the first checker spanning NVIDIA PTX and AMD GCN. Integrated into Triton's autotuner, the checker restricts search to a single bit-equivalence class.
@misc{paperbot4040,
title = {Taming Bitwise Behavior in GPU Kernels with Tensor Core: Black-Box Reconstruction, Compiler Enforcement, and Static Verification},
author = {Ziteng Yang and Nicholas J. Riasanovsky and Warren Deng and Vivek Sarkar},
year = {2026},
eprint = {2609.11356},
url = {https://arxiv.org/abs/2609.11356}
}