一句话：提出三层嵌入向量表查询系统 EVStore，利用结构规律与领域近似优化缓存，显著降低延迟并提升吞吐。
深度推荐系统依赖快速推理，但为缓解类别变量稀疏性，需在内存中保存不断增大的嵌入向量（EV）表。推理具有全有或全无特性：一次请求需多次 EV 表查找，任一内存访问慢都会拖慢整个请求。为此，作者设计、实现并评估 EVStore，一个三层 EV 表查找系统，同时利用推理操作中的结构规律和领域特定近似来实现优化缓存。评估显示，EVStore 将平均延迟降低最多 23%、p90 延迟降低最多 27%，在准确率损失 0.2% 时吞吐量提升至 4 倍；以少量准确率为代价，还可将深度推荐系统内存占用最多降低 94%。
Modern recommendation systems, primarily driven by deep-learning models, depend on fast model inferences to be useful. To tackle the sparsity in the input space, particularly for categorical variables, such inferences are made by storing increasingly large embedding vector (EV) tables in memory. A core challenge is that the inference operation has an all-or-nothing property: each inference requires multiple EV table lookups, but if any memory access is slow, the whole inference request is slow. In our paper, we design, implement and evaluate EVStore, a 3-layer EV table lookup system that harnesses both structural regularity in inference operations and domain-specific approximations to provide optimized caching, yielding up to 23% and 27% reduction on the average and p90 latency while quadrupling throughput at 0.2% loss in accuracy. Finally, we show that at a minor cost of accuracy, EVStore can reduce the Deep Recommendation System (DRS) memory usage by up to 94%, yielding potentially enormous savings for these costly, pervasive systems.
@inproceedings{paperbot10013,
title = {EVStore: Storage and Caching Capabilities for Scaling Embedding Tables in Deep Recommendation Systems},
author = {Daniar H. Kurniawan and Ruipu Wang and Kahfi S. Zulkifli and Fandi A. Wiranata and John Bent and Ymir Vigfusson and Haryadi S. Gunawi},
booktitle = {ASPLOS},
year = {2023},
doi = {10.1145/3575693.3575718}
}