一句话：提出首个利用端侧 NPU 卸载加速 LLM 预填充的推理系统 llm.npu，通过提示、张量和块三级重构降低延迟与能耗。
端侧大语言模型推理因隐私与移动模型发展受关注，但即使十亿级模型在屏幕 UI 理解等任务中仍受预填充高延迟瓶颈。本文提出 llm.npu，首个利用端侧 NPU 卸载降低预填充延迟的 LLM 推理系统。它从三层重构提示与模型：提示级将变长提示划分为固定大小块并保持数据依赖；张量级抽取显著离群值，以低开销在 CPU/GPU 并行执行；块级按硬件亲和性与精度敏感度，将 Transformer 块乱序调度到 CPU/GPU 与 NPU。评估显示，较基线平均预填充加速 22.4 倍、节能 30.7 倍，真实端到端应用最高加速 32.8 倍，并首次在十亿级模型上实现超 1000 tokens/sec 预填充。
On-device inference for Large Language Models (LLMs), driven by increasing privacy concerns and advancements of mobile-sized models, has gained significant interest. However, even mobile-sized LLMs (e.g., Gemma-2B) encounter unacceptably high inference latency, often bottlenecked by the prefill stage in tasks like screen UI understanding. We present llm.npu, the first LLM inference system utilizing on-device Neural Processing Unit (NPU) offloading to reduce prefill latency. llm.npu enhances NPU offloading efficiency by re-constructing the prompt and model in three levels: (1) At prompt level, it divides variable-length prompts into multiple fixed-sized chunks while maintaining data dependencies; (2) At tensor level, it identifies and extracts significant outliers to run on the CPU/GPU in parallel with minimal overhead; (3) At block level, it schedules Transformer blocks in an out-of-order manner to the CPU/GPU and NPU based on their hardware affinity and sensitivity to accuracy. Compared to competitive baselines, llm.npu achieves 22.4x faster prefill speed and 30.7x energy savings on average, and up to 32.8x speedup in an end-to-end real-world application. For the first time, llm.npu achieves more than 1,000 tokens/sec prefilling for a billion-sized model.