一句话：提出边缘 LLM 测试时扩展服务系统 FastTTS，通过投机束扩展、非对称内存分配与动态前缀调度降低延迟并提升吞吐。
边缘 LLM 智能体需本地部署以保护隐私并支持离线，但设备内存有限，只能运行较小模型，推理能力远弱于云端；现有测试时扩展（TTS）方法硬件开销过大，难以实用。本文提出 FastTTS，一个面向内存受限 LLM 推理的高效 TTS 服务系统。作者分析多种 TTS 方法的共性模式与性能瓶颈，设计三项技术：投机束扩展缓解不规则推理路径的长尾；非对称多模型内存分配动态平衡 token 生成与验证的内存；动态前缀感知调度最大化搜索路径间的 KV-cache 复用。FastTTS 以 vLLM 之上的即插即用库实现，使单张 24GB GPU 上的 ≤7B 边缘模型达到云端准确率与延迟。评估显示，相比 vLLM 基线，平均 goodput 提升 2.2 倍，延迟降低 38%-68%。
Recent advances in reasoning Large Language Models (LLMs) are driving the emergence of agentic AI systems. Edge deployment of LLM agents near end users is increasingly necessary to protect data privacy, enable offline use, and provide responsive interaction with local context. However, strict memory constraints on edge devices limit deployment to smaller LLMs, whose reasoning capabilities are much weaker than those of large cloud models, hindering practical deployment of edge agentic AI. Test-Time Scaling (TTS) offers a promising solution by allocating more compute during inference to enhance the reasoning capability of edge LLMs. However, current TTS methods introduce heavy hardware performance overhead on resource-constrained devices, making them impractical for real applications. To address this challenge, we present FastTTS, a serving system that enables fast and efficient TTS for memory-constrained LLM reasoning. After analyzing common patterns across various TTS methods and identifying their performance bottlenecks, we introduce three novel techniques: i) Speculative Beam Extension, which mitigates system stragglers caused by irregular reasoning paths, ii) Asymmetric Multi-Model Memory Allocation, which dynamically balances memory usage between token generation and reasoning-step verification, and iii) Dynamic Prefix-Aware Scheduling, which optimizes reasoning execution to maximize KV-cache reuse across search paths. FastTTS offers a plug-and-play third-party library on top of vLLM, enabling edge LLMs (łeq 7B) on a single consumer GPU (24 GB) to match cloud-model accuracy and cloud-measured latency. Comprehensive evaluation shows that FastTTS achieves an average 2.2× higher goodput and reduces latency by 38%-68% compared to the vLLM baseline; it pushes the boundaries of low-latency Test-Time Scaling on memory-constrained edge devices and highlights the potential for democratizing agentic AI.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{ChenML0MLF26,
title = {FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning},
author = {Hao Mark Chen and Zhiwen Mo and Guanxi Lu and Shuang Liang and Lingxiao Ma and Wayne Luk and Hongxiang Fan},
booktitle = {ASPLOS},
year = {2026},
doi = {10.1145/3779212.3790161}
}