一句话：首次实现实用的 W4A4KV4 LLM 服务，通过细粒度混合精度量化与 GPU 内核优化显著提速。
LLM 服务受权重、激活与 KV cache 开销制约，已有 8 位权重-激活或 4 位仅权重量化对低精度激活支持不足，性能提升有限。本文首次实现实用的 W4A4KV4 服务：提出细粒度混合精度量化 FMPQ，将多数激活压到 4 位且精度损失可忽略；设计优化 W4Ax 内核支持 W4A4/W4A8 混合精度矩阵乘，采用混合精度数据布局、GPU 软件流水线隐藏加载与反量化开销，并通过细粒度 SM 调度实现负载均衡。集成到 COMET 推理框架后支持 LLaMA-3-70B 等模型；在单张 A100-80G-SMX4 上运行 LLaMA 系列，内核较 cuBLAS 提速 2.88 倍，吞吐较 TensorRT-LLM 提升 2.02 倍。
Quantization is a widely-used compression technology to reduce the overhead of serving large language models (LLMs) on terminal devices and in cloud data centers. However, prevalent quantization methods, such as 8-bit weight-activation or 4-bit weight-only quantization, achieve limited performance improvements due to poor support for low-precision (e.g., 4-bit) activation. This work, for the first time, realizes practical W4A4KV4 serving for LLMs, fully utilizing the INT4 tensor cores on modern GPUs and reducing the memory bottleneck caused by the KV cache. Specifically, we propose a novel fine-grained mixed-precision quantization algorithm (FMPQ) that compresses most activations into 4-bit with negligible accuracy loss. To support mixed-precision matrix multiplication for W4A4 and W4A8, we develop a highly optimized W4Ax kernel. Our approach introduces a novel mixed-precision data layout to facilitate access and fast dequantization for activation and weight tensors, utilizing the GPU's software pipeline to hide the overhead of data loading and conversion. Additionally, we propose fine-grained streaming multiprocessor (SM) scheduling to achieve load balance across different SMs. We integrate the optimized W4Ax kernel into our inference framework, COMET, and provide efficient management to support popular LLMs such as LLaMA-3-70B. Extensive evaluations demonstrate that, when running LLaMA family models on a single A100-80G-SMX4, COMET achieves a kernel-level speedup of 2.88x over cuBLAS and a 2.02x throughput improvement compared to TensorRT-LLM from an end-to-end framework perspective.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{Liu0RXPWL0W25,
title = {COMET: Towards Practical W4A4KV4 LLMs Serving},
author = {Lian Liu and Long Cheng and Haimeng Ren and Zhaohui Xu and Yudong Pan and Mengdi Wang and Xiaowei Li and Yinhe Han and Ying Wang},
booktitle = {ASPLOS},
year = {2025},
doi = {10.1145/3676641.3716252}
}