一句话：提出 Superchip 中心的 offloading 系统 SuperOffload，显著提升大规模 LLM 训练吞吐并支持超长序列。
Superchip 将 GPU 与 CPU 紧耦合于同一封装，算力空前，但其对 LLM 训练的收益尚缺研究。本文首次研究面向 Superchip 的 offloading 训练方案，指出其与传统松耦合架构的差异要求重新审视既有 offloading 假设。提出的 SuperOffload 以 Superchip 为中心，高效利用 Hopper GPU、Grace CPU 与 NVLink-C2C，结合自适应权重 offloading、bucketization 重划分、Superchip-aware casting、推测执行及 Grace CPU 优化 Adam。在 NVIDIA GH200 上，其吞吐较最先进 offloading 系统最高提升 2.5×，单 Superchip 可训练 25B 模型；扩展 ZeRO 数据并行与 DeepSpeed-Ulysses 序列并行后，在 8 张 GH200 上以 55% MFU 训练序列长度达 100 万 token 的 13B 模型。
The emergence of Superchips represents a significant advancement in next-generation AI hardware. These Superchips employ a tightly coupled heterogeneous architecture that integrates GPU and CPU on the same package, which offers unprecedented computational power. However, there has been scant research investigating how LLM training benefits from this new architecture. In this work, for the first time, we study LLM training solutions based on offloading for Superchips. We observe important differences between Superchips and traditional loosely-coupled GPU-CPU architecture, which necessitate revisiting prevailing assumptions about offloading. Based on that, we present SuperOffload, a Superchip-centric offloading system that simultaneously uses Hopper GPU, Grace CPU, and NVLink-C2C interconnect more efficiently. SuperOffload accomplishes this via a combination of techniques, such as adaptive weight offloading, bucketization repartitioning, Superchip-aware casting, speculative execution, and a highly optimized Adam optimizer for Grace CPUs. Our evaluation of SuperOffload on NVIDIA GH200 demonstrates up to 2.5× throughput improvement compared to state-of-the-art offloading-based systems, enabling training of up to 25B model on a single Superchip while achieving high training throughput. We also extend SuperOffload with ZeRO-style data parallelism and DeepSpeed-Ulysses sequence parallelism, enabling training of 13B model with sequence lengths up to 1 million tokens on 8 GH200 while achieving 55% MFU.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{LianTRZ26,
title = {SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips},
author = {Xinyu Lian and Masahiro Tanaka and Olatunji Ruwase and Minjia Zhang},
booktitle = {ASPLOS},
year = {2025},
doi = {10.1145/3760250.3762217}
}