一句话：提出 Two-Face，按稀疏矩阵的稠密/稀疏区域混合集合通信与细粒度单边通信，显著加速分布式 SpMM。
稀疏矩阵-稠密矩阵乘法（SpMM）广泛用于科学计算与图神经网络。分布式执行中通信开销往往占主导：稀疏无关的集合通信会传输冗余数据，而细粒度稀疏感知通信又因软件开销高而低效。论文观察到稀疏矩阵通常兼具较稠密与较稀疏区域，据此将通信划分为稀疏无关和稀疏感知两部分，并提出 Two-Face 算法：稠密区域使用集合通信，稀疏区域使用细粒度单边通信。在 4096 核超级计算机上，Two-Face 相较先前工作平均加速 2.11 倍，并能随机器规模良好扩展。
Sparse matrix dense matrix multiplication (SpMM) is commonly used in applications ranging from scientific computing to graph neural networks. Typically, when SpMM is executed in a distributed platform, communication costs dominate. Such costs depend on how communication is scheduled. If it is scheduled in a sparsity-unaware manner, such as with collectives, execution is often inefficient due to unnecessary data transfers. On the other hand, if communication is scheduled in a fine-grained sparsity-aware manner, communicating only the necessary data, execution can also be inefficient due to high software overhead. We observe that individual sparse matrices often contain regions that are denser and regions that are sparser. Based on this observation, we develop a model that partitions communication into sparsity-unaware and sparsity-aware components. Leveraging the partition, we develop a new algorithm that performs collective communication for the denser regions, and fine-grained, one-sided communication for the sparser regions. We call the algorithm Two-Face. We show that Two-Face attains an average speedup of 2.11x over prior work when evaluated on a 4096-core supercomputer. Additionally, Two-Face scales well with the machine size.
@inproceedings{BlockGMAT24,
title = {Two-Face: Combining Collective and One-Sided Communication for Efficient Distributed SpMM},
author = {Charles Block and Gerasimos Gerogiannis and Charith Mendis and Ariful Azad and Josep Torrellas},
booktitle = {ASPLOS},
year = {2024},
doi = {10.1145/3620665.3640427}
}