一句话：提出 XY-Serve，以细粒度 meta 原语抽象与虚拟填充抹平 LLM 动态负载，在昇腾 NPU 上大幅提升服务吞吐。
生产级 LLM 服务需要兼顾低延迟与成本效率，但 LLM 输入输出长度动态不可预测，叠加各类优化后进一步放大了工作负载的变化性，使 AI 加速器（尤其是基于 tile 编程模型的 DSA）难以维持高效执行。本文提出 XY-Serve，一个昇腾 NPU 原生的端到端 LLM 服务系统。其核心是一种抽象机制：将计算分解为统一、硬件友好、细粒度的 meta 原语，从而抹平负载不规则性，使 kernel 无需感知动态形状即可高效执行。在此之上，针对 Attention 提出以架构感知 tile 尺寸计算 GEMM-Softmax-GEMM 基本模式的 meta-kernel；针对 Linear 提出虚拟填充方案，在适配动态形状变化的同时复用多种固定 tile 尺寸的高效 GEMM 原语。XY-Serve 与 vLLM 兼容。实验表明，在昇腾 NPU 上端到端吞吐相比公开基线最高提升 95%，Linear 与 Attention kernel 分别平均提速 14.6% 与 21.5%，并在 GPU 平台验证了通用性。
Meeting growing demands for low latency and cost efficiency in production-grade large language model (LLM) serving systems requires integrating advanced optimization techniques. However, dynamic and unpredictable input-output lengths of LLM, compounded by these optimizations, exacerbate the issues of workload variability, making it difficult to maintain high efficiency on AI accelerators, especially DSAs with tile-based programming models. To address this challenge, we introduce XY-Serve, a versatile, Ascend NPU native, end-to-end production LLM-serving system. The core idea is an abstraction mechanism that smooths out the workload variability by decomposing computations into unified, hardware-friendly, fine-grained meta primitives. Then, kernels can efficiently execute without concerning the irregularity of workload. After this abstraction mechanism, for Attention, we propose a meta-kernel that computes the basic pattern of GEMM-Softmax-GEMM with architectural-aware tile sizes. For Linear, we introduce a virtual padding scheme that adapts to dynamic shape changes while using highly efficient GEMM primitives with assorted fixed tile sizes. XY-Serve sits harmoniously with vLLM. Experimental results show up to 95% end-to-end throughput improvement compared with current publicly available baselines on Ascend NPUs. We also set a new performance record for Linear (average 14.6% faster) and Attention (average 21.5% faster) kernels relative to existing libraries. Lastly, we demonstrate the generality of our technologies on GPU platform.
@inproceedings{SongTHLWMXSJY0L26,
title = {XY-Serve: End-to-End Versatile Production Serving for Dynamic LLM Workloads},
author = {Mingcong Song and Xinru Tang and Fengfan Hou and Jing Li and Wei Wei and Yipeng Ma and Runqiu Xiao and Hongjie Si and Dingcheng Jiang and Shouyi Yin and Yang Hu and Guoping Long},
booktitle = {ASPLOS},
year = {2025},
doi = {10.1145/3760250.3762228}
}