一句话：通过软硬件协同设计，透明地将生产者计算与后续通信融合，减少资源争用并高效重叠串行化通信。
大语言模型的训练与推理日益依赖分布式技术，设备间通信随规模增大而降低扩展效率。虽然部分通信可与独立计算重叠隐藏，但张量并行等技术的通信与模型执行本质串行。软件层面细粒度交错通信与计算实现困难，且并发执行会引发计算与通信对计算、内存资源的争用，削弱重叠效果。T3 提出软硬件协同设计：通过配置生产者输出地址空间透明地将生产者操作与后续通信融合，仅需少量软件改动；硬件上加入轻量级 track-and-trigger 机制编排生产者计算与通信，并利用 compute-enhanced memory 承担通信伴随计算。在 T-NLG 等 Transformer 模型上，T3 将通信密集子层加速 30% 几何均值（最高 47%），减少数据移动 22%（最高 36%）；在约 5000 亿参数的 PALM 与 MT-NLG 上仍有 29% 几何均值收益。
Large Language Models increasingly rely on distributed techniques for their training and inference. These techniques require communication across devices which can reduce scaling efficiency as the number of devices increases. While some distributed techniques can overlap, and thus, hide this communication with independent computations, techniques such as Tensor Parallelism (TP) inherently serialize communication with model execution. One approach to hide this serialized communication is to interleave it with the producer operation (of the communicated data) in a fine-grained manner. However, this fine-grained interleaving of communication and computation in software can be difficult. Furthermore, as with any concurrent execution, it requires compute and memory resources to be shared between computation and communication, causing resource contention that reduces overlapping efficacy. To overcome these challenges, we propose T3 which applies hardware-software co-design to transparently overlap serialized communication while minimizing resource contention with compute. T3 transparently fuses producer operations with the subsequent communication via a simple configuration of the producer's output address space and requires minor software changes. At the hardware level, T3 adds a lightweight track and trigger mechanism to orchestrate the producer's compute, and communication. It further uses compute-enhanced memories for communication's attendant compute. As a result, T3 reduces resource contention, and efficiently overlaps serialized communication with computation. For important Transformer models like T-NLG, T3 speeds up communication-heavy sublayers by 30% geomean (max 47%) and reduces data movement by 22% geomean (max 36%). Furthermore, T3's benefits persist as models scale: geomean 29% for sublayers in ~500-billion parameter models, PALM and MT-NLG.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2401-16677,
title = {T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives},
author = {Suchita Pati and Shaizeen Aga and Mahzabeen Islam and Nuwan Jayasena and Matthew D. Sinclair},
booktitle = {ASPLOS},
year = {2024},
doi = {10.1145/3620665.3640410}
}