一句话：提出归约融合算法，借助逆向重计算将依赖的 reducer 与 mapper 融合为统一 reducer，最高加速 2.47×。
分布式数据并行计算是大数据应用与新兴大语言模型任务的关键基础，其效率很大程度上取决于 reducer 的性能，尤其是在面对大规模数据访问时。本文提出一种新的归约融合（reduction fusion）算法，通过将相互依赖的 reducer 与 mapper 融合为一个统一的 reducer 来优化分布式数据并行程序。该算法采用逆向重计算（inverse recomputation）技术，在融合过程中保留部分聚合结果，从而降低存储、网络 I/O、内存与缓存开销。初步评估显示，该方法可带来最高 2.47 倍的性能提升，表明其在实际分布式计算环境中的有效性与可行性，也展示了应对大规模数据访问挑战的潜力。
Distributed data-parallel computations are critical for both traditional big data applications and emerging large language model tasks. The efficiency of these computations largely depends on reducer performance, particularly in handling extensive data access. This paper introduces a novel reduction fusion algorithm that optimizes distributed data-parallel programs by fusing dependent reducers and mappers into a single, unified reducer. Employing inverse recomputation, the algorithm preserves partial aggregation and reduces storage, network I/O, memory, and cache overheads. Our preliminary evaluation reveals performance improvements of up to 2.47×, demonstrating the practicality and effectiveness of this approach, while also highlighting its potential to address challenges posed by extensive data access in modern distributed computing environments.
@inproceedings{paperbot8952,
title = {Reduction Fusion for Optimized Distributed Data-Parallel Computations via Inverse Recomputation},
author = {Haoxiang Lin and Yang Wang and Yanjie Gao and Hongyu Zhang and Ming Wu and Mao Yang},
booktitle = {FSE},
year = {2025},
doi = {10.1145/3696630.3728496}
}