一句话：提出 DSV，利用动态注意力稀疏与稀疏感知上下文并行，加速视频 DiT 训练并保持模型质量。
视频 DiT 训练中 3D 全注意力呈二次复杂度，可占 95% 处理时间，成为高清长视频扩展的瓶颈。本文提出 DSV：先用基于低秩近似的两阶段算法捕捉动态注意力稀疏模式，再通过定制内核高效筛选关键 key-value 对并计算稀疏注意力；同时设计稀疏感知混合上下文并行，以在注意力头和块之间重新平衡因稀疏异质性造成的负载倾斜。实验表明，DSV 训练吞吐最高提升 3.02 倍，可扩展至 128 GPU 与 520k token 长度，且模型质量与全注意力方案相当。
Diffusion Transformers (DiTs) have shown remarkable performance in generating high-quality videos. However, the quadratic complexity of 3D full attention remains a bottleneck in scaling DiT training, especially with high-definition, lengthy videos, where it can consume up to 95% of processing time and demand specialized context parallelism. This paper introduces DSV to accelerate video DiT training by leveraging the dynamic attention sparsity we empirically observe. DSV uses a two-stage algorithm to capture the dynamic sparsity patterns via low-rank based approximation of the original query and key. It employs custom kernels to efficiently identify critical key-value pairs and compute the sparse attention. To accommodate the new sparsity dimension, DSV adopts a hybrid sparsity-aware context parallelism that re-balances the skewed workload across attention heads and blocks due to sparsity heterogeneity. DSV achieves up to 3.02x higher training throughput, scaling to 128 GPUs and 520k token lengths, while maintaining model quality comparable to full attention schemes.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{0004CJ00DZJ026,
title = {Dynamic Sparsity in Large-Scale Video DiT Training},
author = {Xin Tan and Yuetao Chen and Yimin Jiang and Xing Chen and Kun Yan and Nan Duan and Yibo Zhu and Daxin Jiang and Hong Xu},
booktitle = {ASPLOS},
year = {2025},
doi = {10.1145/3760250.3762216}
}