一句话：提出因果同步并行机制，实现高性能且可复现的流水线并行超网训练。
超级网训练是神经架构搜索中的重要范式：它把整个 DNN 架构搜索空间嵌入一个单体超网，逐批激活子网进行拟合，再搜索满足特定要求的高质量子网。为加速训练，人们希望在多 GPU 上并行训练子网，但并发子网可能访问相同的 DNN 层，从而产生竞争风险。现有系统既不能高效并行化子网训练执行，也不能确定性地消除该竞争风险，导致训练过程不可复现并可能带来不可忽视的精度损失。NASPipe 提出因果同步并行（causal synchronous parallelism）来支持流水线并行超网训练，在保持高性能的同时实现确定性、可复现的训练，并避免上述精度损失。
Supernet training, a prevalent and important paradigm in Neural Architecture Search, embeds the whole DNN architecture search space into one monolithic supernet, iteratively activates a subset of the supernet (i.e., a subnet) for fitting each batch of data, and searches a high-quality subnet which meets specific requirements. Although training subnets in parallel on multiple GPUs is desirable for acceleration, there inherently exists a race hazard that concurrent subnets may access the same DNN layers. Existing systems support neither efficiently parallelizing subnets’ training executions, nor resolving the race hazard deterministically, leading to unreproducible training procedures and potentiallly non-trivial accuracy loss.
@inproceedings{ZhaoLCSCWZLC22,
title = {NASPipe: high performance and reproducible pipeline parallel supernet training via causal synchronous parallelism},
author = {Shixiong Zhao and Fanxin Li and Xusheng Chen and Tianxiang Shen and Li Chen and Sen Wang and Nicholas Zhang and Cheng Li and Heming Cui},
booktitle = {ASPLOS},
year = {2022},
doi = {10.1145/3503222.3507735}
}