一句话：提出上下文感知的 CodeLLM 驱逐策略 CACE，较 SOTA 降低 70% 首 token 延迟并减少 55% 换出次数。
为兼顾隐私、延迟与模型定制，企业常自托管 CodeLLM，但模型种类与数量增长、加速器内存有限，给模型管理与服务效率带来挑战。本文提出 CACE，一种面向资源受限自托管场景的上下文感知模型驱逐策略。与仅依据近期使用情况的 LRU 等传统策略不同，CACE 综合模型加载时间、任务延迟敏感度、预期输出长度，以及通过滑动窗口跟踪的近期使用与未来需求等多维因素做驱逐决策。作者在包含延迟敏感的代码补全与吞吐密集的代码推理的真实工作负载上评估 CACE，结果显示其将首 token 时间（TTFT）降低 70%、端到端延迟降低 37%，同时模型驱逐次数较最先进系统减少 55%；消融实验进一步说明多因素驱逐对兼顾响应性与资源效率的必要性。
AI-assisted coding tools powered by Code Large Language Models (CodeLLMs) are increasingly integrated into modern software development workflows. To address concerns around privacy, latency, and model customization, many enterprises opt to self-host these models. However, the diversity and growing number of CodeLLMs, coupled with limited accelerator memory, introduce practical challenges in model management and serving efficiency. This paper presents CACE, a novel context-aware model eviction strategy designed specifically to optimize self-hosted CodeLLM serving under resource constraints. Unlike traditional eviction strategies based solely on recency (e.g., Least Recently Used), CACE leverages multiple context-aware factors, including model load time, task-specific latency sensitivity, expected output length, and recent usage and future demand tracked through a sliding window. We evaluate CACE using realistic workloads that include both latency-sensitive code completion and throughput-intensive code reasoning tasks. Our experiments show that CACE reduces Time-to-First-Token (TTFT) by 70% and end-to-end (E2E) latency by 37%, while significantly lowering the number of model evictions by 55% compared to state-of-the-art systems. Ablation studies further demonstrate the importance of multi-factor eviction in balancing responsiveness and resource efficiency. This work contributes practical strategies for deploying scalable, low-latency AI coding assistants in real-world software engineering environments.
DOI 原文 ·
@inproceedings{paperbot9479,
title = {Context-Aware CodeLLM Eviction for AI-assisted Coding},
author = {Kishanthan Thangarajah and Boyuan Chen and Shi Chang and Ahmed E. Hassan},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00271}
}