一句话：提出 Split-CNN 将输入图像切块独立卷积以大幅降低训练显存，并配套异构内存管理系统提升 GPU 训练可扩展性。
本文针对深度卷积神经网络训练中的内存瓶颈展开跨层研究。作者首先提出 Split Convolutional Neural Network（Split-CNN）：通过对当前主流 CNN 模型进行自动变换得到，其核心区别在于把输入图像切分成小块，在进入模型后续阶段之前对这些小块独立地进行卷积运算，从而显著降低训练算法对显存的需求。其次，作者设计了一种异构内存管理系统（HMMS），以充分利用 Split-CNN 在访存上的友好特性。实验表明，Split-CNN 大幅减少了 GPU 加速器上的训练内存占用，带来更高的训练可扩展性；此外，实证结果显示，在随机选取的边界处进行切分甚至能因正则化效应而在精度上超过基线 CNN。
We present an interdisciplinary study to tackle the memory bottleneck of training deep convolutional neural networks (CNN). Firstly, we introduce Split Convolutional Neural Network (Split-CNN) that is derived from the automatic transformation of the state-of-the-art CNN models. The main distinction between Split-CNN and regular CNN is that Split-CNN splits the input images into small patches and operates on these patches independently before entering later stages of the CNN model. Secondly, we propose a novel heterogeneous memory management system (HMMS) to utilize the memory-friendly properties of Split-CNN. Through experiments, we demonstrate that Split-CNN achieves significantly higher training scalability by dramatically reducing the memory requirements of training algorithms on GPU accelerators. Furthermore, we provide empirical evidence that splitting at randomly chosen boundaries can even result in accuracy gains over baseline CNN due to its regularization effect.
DOI 原文 ·
@inproceedings{paperbot4883,
title = {Split-CNN},
author = {Tian Jin and Seokin Hong},
booktitle = {ASPLOS},
year = {2019},
doi = {10.1145/3297858.3304038}
}