一句话：提出 SNIP：利用前向损失偏差与后向权重偏差指标，通过 ILP 逐层自适应分配 subbyte 混合精度，保持 LLM 质量并最多减少 80% FLOPs。
大语言模型训练中，现代 GPU 支持 subbyte 精度，但如何在不损害模型质量的前提下高效利用该精度仍具挑战。现有混合精度训练要么对所有 GEMM 统一精度，要么依赖难以泛化的启发式方法，导致收敛次优甚至不稳定。本文提出 SNIP，一种细粒度自适应混合精度训练框架，用于支持 subbyte 精度的 LLM 预训练。SNIP 周期性收集激活、梯度和优化器状态统计，定义前向传播中的损失偏差（量化导致训练损失上升）与反向传播中的权重偏差（梯度误差传播影响更新）两个指标，并基于这些指标构建整数线性规划问题，在满足效率目标的同时逐层优化精度以最小化整体质量损失。在 1B、3B、7B 和 70B 的 Llama 类模型上，SNIP 持续优于现有基线，最多减少 80% FLOPs，并在不同模型规模和训练阶段保持模型质量，计算开销极小。
Training large language models (LLMs) efficiently while preserving model quality poses significant challenges, particularly with subbyte precision supported by state-of-the-art GPUs. Current mixed-precision training approaches either apply uniform precision to all GEMM operations or rely on heuristic-based methods that fail to generalize during training, leading to suboptimal convergence and instability. To address these challenges, this paper introduces SNIP, a fine-grained adaptive mixed-precision training framework for LLM pretraining that supports subbyte precision. SNIP periodically collects statistics on activations, gradients, and optimizer states to assess the precision loss impact on model quality. We define two key metrics: loss divergence in the forward pass, caused by quantization-induced increases in training loss, and weight divergence in the backward pass, which measures error propagation through gradients affecting model updates. These metrics guide an Integer Linear Programming (ILP) problem that systematically optimizes layerwise precision to minimize overall quality loss while meeting efficiency targets. Experiments on 1B, 3B, 7B and 70B Llama-like models demonstrate that SNIP consistently outperforms existing baselines, reducing FLOPs by up to 80% while preserving model quality across different model sizes and training phases with minimal computational overhead.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{PanYYM26,
title = {SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Training},
author = {Yunjie Pan and Yongyi Yang and Hanmei Yang and Scott Mahlke},
booktitle = {ASPLOS},
year = {2026},
doi = {10.1145/3779212.3790223}
}