一句话：揭示反向模式自动微分与定界延续（shift/reset）的联系，并用多阶段编程实现兼顾性能与表达力的可微编程。
随着神经网络架构日益复杂多样，人们希望支持更一般的可微编程：任意带参数的计算都能通过梯度下降训练。本文重新审视自动微分（AD）技术，重点解释反向模式 AD 及其对神经网络反向传播的推广。作者发现反向模式 AD 与定界延续（delimited continuations）之间存在紧密联系，从而可以仅通过运算符重载实现反向 AD，无需维护辅助数据结构。进一步，将这一形式与多阶段编程（staging）结合，得到一种高效实现，兼具基于显式具体化计算图的框架（如 TensorFlow）的性能优势和纯库方法（如 PyTorch）的表达力。
Deep learning has seen tremendous success over the past decade in computer vision, machine translation, and gameplay. This success rests crucially on gradient-descent optimization and the ability to “learn” parameters of a neural network by backpropagating observed errors. However, neural network architectures are growing increasingly sophisticated and diverse, which motivates an emerging quest for even more general forms of differentiable programming, where arbitrary parameterized computations can be trained by gradient descent. In this paper, we take a fresh look at automatic differentiation (AD) techniques, and especially aim to demystify the reverse-mode form of AD that generalizes backpropagation in neural networks. We uncover a tight connection between reverse-mode AD and delimited continuations, which permits implementing reverse-mode AD purely via operator overloading and without managing any auxiliary data structures. We further show how this formulation of AD can be fruitfully combined with multi-stage programming (staging), leading to an efficient implementation that combines the performance benefits of deep learning frameworks based on explicit reified computation graphs (e.g., TensorFlow) with the expressiveness of pure library approaches (e.g., PyTorch).
@article{paperbot608,
title = {Demystifying differentiable programming: shift/reset the penultimate backpropagator},
author = {Fei Wang and Daniel Zheng and James Decker and Xilun Wu and Grégory M. Essertel and Tiark Rompf},
journal = {Proceedings of the ACM on Programming Languages},
volume = {3},
number = {ICFP},
year = {2019},
doi = {10.1145/3341700}
}