一句话：提出感知 3D 并行的通信压缩方案 Optimus-CC，在降低跨节点通信量的同时避免模型质量过度损失。
训练现代大型 NLP 模型通常采用 3D 并行将模型切分到多张 GPU 上，但由此带来的跨节点通信开销很高。压缩通信流量是缓解该开销的常见思路，然而现有压缩技术难以适用于 3D 并行的 NLP 训练：一方面只针对数据并行产生的通信，另一方面其压缩方案本身已严重损害模型质量。针对这些局限，作者提出 Optimus-CC，一种感知 3D 并行的通信压缩方法，目标是在降低跨节点通信量的同时保持模型质量。
In training of modern large natural language processing (NLP) models, it has become a common practice to split models using 3D parallelism to multiple GPUs. Such technique, however, suffers from a high overhead of inter-node communication. Compressing the communication is one way to mitigate the overhead by reducing the inter-node traffic volume; however, the existing compression techniques have critical limitations to be applied for NLP models with 3D parallelism in that 1) only the data parallelism traffic is targeted, and 2) the existing compression schemes already harm the model quality too much.
@inproceedings{paperbot10037,
title = {Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression},
author = {Jaeyong Song and Jinkyu Yim and Jaewon Jung and Hongsun Jang and Hyung-Jin Kim and Youngsok Kim and Jinho Lee},
booktitle = {ASPLOS},
year = {2023},
doi = {10.1145/3575693.3575712}
}