一句话：提出 Shift Parallelism，动态切换张量并行与序列并行，兼顾 LLM 推理低延迟与高吞吐。
大语言模型推理需同时实现低延迟与高吞吐。张量并行（TP）可降低响应延迟，但 GPU 通信拖累合并 token 吞吐；数据并行（DP）吞吐更高却延迟较差。由于不同并行方式下 KV cache 存在差异，TP 与 DP 难以直接结合。本文观察到序列并行（SP，训练中的 Ulysses）具有类似 DP 的性质，但保持 KV cache 不变性，因而可适配推理并与 TP 结合。作者提出 Shift Parallelism：根据工作负载动态在 TP 与 SP 间切换，在低流量时最小化延迟，在高流量时不损失吞吐。基于真实生产 trace 和合成负载，覆盖不同模型、上下文长度与到达率，结果显示交互负载响应最高快 1.51 倍，批处理吞吐最高提升 50%，相比仅 TP 方案获得更好的延迟-吞吐权衡。
Efficient parallelism is necessary for achieving low-latency, high-throughput inference with large language models (LLMs). Tensor parallelism (TP) is the state-of-the-art method for reducing LLM response latency, however GPU communications reduces combined token throughput. On the other hand, data parallelism (DP) obtains a higher throughput yet is slow in response latency. Best of both worlds does not exist, and it is not possible to combine TP and DP because of the KV cache variance across the parallelisms. We notice Sequence Parallelism (SP—Ulysses in training) has similar properties as DP but with KV cache invariance. We adapt SP to inference, and combine it with TP to get the best of both worlds. Our solution: Shift Parallelism. Shift Parallelism dynamically switches across TP and SP, and minimizes latency in low traffic without losing throughput in high traffic. The efficient GPU communications of Shift Parallelism yields up to i) 1.51x faster response in interactive workloads and ii) 50% higher throughput in batch workloads, compared to a TP-only solution. We evaluate Shift Parallelism with real-world production traces with dynamic traffic patterns as well as synthetic benchmarking patterns across models, context sizes, and arrival rates. All results affirm the same: Shift Parallelism has a better latency vs. throughput tradeoff than TP or DP, and hence obtains low latency without degrading throughput in dynamic workloads.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2509-16495,
title = {Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads},
author = {Mert Hidayetoglu and Aurick Qiao and Michael Wyatt and Jeff Rasley and Yuxiong He and Samyam Rajbhandari},
booktitle = {ASPLOS},
year = {2026},
doi = {10.1145/3779212.3790219}
}