Although there have been significant efforts to make LLM serving efficient, we observe two limitations of current state-of-the-art serving frameworks in handling multi-turn dialogues between users and assistants, particularly in chat scenarios. First, existing LLM frameworks incur substantial computational overhead in recomputing attention keys and values (KVs) for understanding context across multiple turns of user queries. Second, as the prompt length of user queries is amplified due to multi-turns, a first-come-first-served (FCFS) scheduling policy often causes head-of-line blocking issues, leading to underutilization of GPU resources. To address these limitations, we present FlashGen to rapidly complete multi-turn queries by efficiently utilizing the compute and memory resources of GPUs as well as the host hardware (e.g., DRAM and SSD). We introduce a multi-level KV cache comprised of GPU, CPU, and SSD, to efficiently retain attention KVs from prior turns. Our approach employs low-cost cache restoration techniques to avoid the recomputation burden. Further, we propose a request reordering technique to effectively utilize GPU memory. This scheduling technique carefully adjusts the request order without compromising fairness. Our proposed techniques outperform the vLLM framework in terms of both latency and throughput. For OPT 30B and Llama-2 70B models with the ShareGPT dataset, we achieve 1.63x and 2.85x better throughput, respectively while in a similar latency boundary.

@inproceedings{JeongA25,
  title = {Accelerating LLM Serving for Multi-turn Dialogues with Efficient Resource Management},
  author = {Jinwoo Jeong and Jeongseob Ahn},
  booktitle = {ASPLOS},
  year = {2025},
  doi = {10.1145/3676641.3716245}
}