一句话：提出以原语为中心的 GPU 集合通信库性能分析与工具 CCLInsight，揭示配置规律并显著优化 NCCL/MSCCL。
随着分布式 AI 工作负载规模增长，NCCL 等 GPU 集合通信库（xCCL）对异构硬件上的高效数据移动愈发关键。既有分析常缺乏深度、通用性或优化洞见。本文提出以原语为中心的 profiling 与分析方法，整合体系结构、通信算法和底层原语配置，全面刻画性能关键参数，并实现工具 CCLInsight。作者在 64 H100、64 RTX 5000、256 A100 三个集群进行设计分析，在 256 H200 集群进行应用评估，揭示参数对通信效率与扩展性的影响，并提出 collapsed optimal configuration scaling law。按该分析调参后，微基准中 NCCL 提升 43.90%，MSCCL 加速 40.64×；在 GPT-3、LLaMA2、DeepSeek-R1 训练中，NCCL 与 MSCCL 分别最多提升 2.27× 和 3.00×。
As distributed AI workloads scale in complexity, GPU-based collective communication libraries (xCCLs) such as NCCL are becoming increasingly critical for efficient data movement across heterogeneous hardware. Characterizing and optimizing the interactions between hardware architectures, communication algorithm designs, and low-level primitive configurations is essential to fully utilize GPU and interconnect resources while ensuring scalability. However, existing analysis methods often lack depth, limit optimization insights, leave significant performance gains untapped, or lack generality. To address these limitations, we propose a novel primitive-centric profiling and analysis method that integrates architectural, algorithmic, and primitive-level insights to comprehensively qualify and quantify performance-critical parameters in GPU-based CCLs. We implement and evaluate this method in our tool CCLInsight. Evaluating three large-scale GPU clusters with diverse GPU architectures (64 H100s, 64 RTX 5000s, and 256 A100s) for design analysis, and a fourth 256-GPU H200 cluster for application-level evaluation, CCLInsight uncovers critical parameter influences on communication efficiency and scalability, as well as exposing a new CCL performance scaling law, the collapsed optimal configuration scaling law. Adjusting parameters based on this analysis, we demonstrate a 43.90% improvement in NCCL and a 40.64× speedup in MSCCL over default configurations in the collective communication microbenchmark, while also improving NCCL and MSCCL performance by up to 2.27× and 3.00×, respectively, on LLM training workloads (GPT-3, LLaMA2, DeepSeek-R1). These contributions establish CCLInsight as a useful tool for xCCL characterization and optimization in distributed AI environments.
@inproceedings{paperbot11734,
title = {CCLInsight: Unveiling Insights in GPU Collective Communication Libraries via Primitive-Centric Analysis},
author = {Liuyao Dai and Adam Weingram and Weicong Chen and Xiaoyi Lu},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3773105}
}