一句话：提出 LazyDP，通过算法-软件协同设计消除 DP-SGD 训练推荐系统的计算与内存瓶颈，平均加速 119 倍且保持数学等价。
差分隐私（DP）已被工业界广泛用于隐私保护，但针对推荐系统（RecSys）的私有训练计算挑战尚未被充分研究。本文首先刻画使用 DP-SGD 训练私有 RecSys 的性能瓶颈，发现噪声采样阶段受计算限制、带噪梯度更新阶段受内存带宽限制，导致显著开销。为此，作者提出 LazyDP，一种算法-软件协同设计，针对上述计算与内存挑战进行优化。与最先进的 DP-SGD 训练系统相比，LazyDP 平均取得 119 倍训练吞吐提升，同时保证训练出的差分隐私推荐模型在数学上等价。
Differential privacy (DP) is widely being employed in the industry as a practical standard for privacy protection. While private training of computer vision or natural language processing applications has been studied extensively, the computational challenges of training of recommender systems (RecSys) with DP have not been explored. In this work, we first present our detailed characterization of private RecSys training using DP-SGD, root-causing its several performance bottlenecks. Specifically, we identify DP-SGD's noise sampling and noisy gradient update stage to suffer from a severe compute and memory bandwidth limitation, respectively, causing significant performance overhead in training private RecSys. Based on these findings, we propose LazyDP, an algorithm-software co-design that addresses the compute and memory challenges of training RecSys with DP-SGD. Compared to a state-of-the-art DP-SGD training system, we demonstrate that LazyDP provides an average 119× training throughput improvement while also ensuring mathematically equivalent, differentially private RecSys models to be trained.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{LimKHMSR24,
title = {LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Models},
author = {Juntaek Lim and Youngeun Kwon and Ranggi Hwang and Kiwan Maeng and Edward Suh and Minsoo Rhu},
booktitle = {ASPLOS},
year = {2024},
doi = {10.1145/3620665.3640384}
}