一句话:将反向传播范畴化:证明梯度下降是从参数化函数到更新规则范畴的 monoidal functor,并引入 request function。
监督学习算法在由参数空间 P 参数化的函数集 A→B 中搜索,以逼近理想函数 f:A→B,并依据样例 (a,f(a)) 更新参数。本文定义一个更新规则可组合的范畴,把参数化函数视为对象/态射,更新规则视为可组合过程。核心贡献是 request function,用结构化方式描述反向传播:训练时先请求误差信息,再反向传播更新。作者证明,在固定步长和满足特定性质的误差函数下,梯度下降定义了一个从参数化函数范畴到更新规则范畴的 monoidal functor。该视角推广了神经网络,并把反向传播与双向编程和 open games 中的结构联系起来。
A supervised learning algorithm searches over a set of functions$A\rightarrow B$parametrised by a space$P$to find the best approximation to some ideal function$f:A\rightarrow B$. It does this by taking examples$(a, f(a))\in A\times B$, and updating the parameter according to some rule. We define a category where these update rules may be composed, and show that gradient descent-with respect to a fixed step size and an error function satisfying a certain property-defines a monoidal functor from a category of parametrised functions to this category of update rules. A key contribution is the notion of request function. This provides a structural perspective on backpropagation, giving a broad generalisation of neural networks and linking it with structures from bidirectional programming and open games.