一句话:提出 PyPar,自动发现真实 Python 程序的并行化机会,并生成并行化报告与参考并行实现。
Python 程序常存在未被开发者利用的并行化机会,现有自动并行化方法多局限于特定领域或需手工标注,难以通用。本文提出 PyPar:先通过数据依赖分析判断代码能否并发执行,核心是图论方法;再用动态筛选策略剔除低效并行;最后输出并行性报告,并基于线程、进程或 Ray 生成参考并行化程序。作者实现原型并在 Scikit-Image、SciPy、librosa、trimesh、Scikit-learn 和 seaborn 六个真实包上评估,共测试 1,240 个函数,发现 127 个可并行函数;人工过滤后仅 7 个误报,精度 94.5%。120 个可并行函数约占测试函数 10%,多数可获最高 90%、平均 44% 的加速,接近理论估计。
Parallelization is a promising way to improve the performance of Python programs. Unfortunately, developers may miss parallelization possibilities, because they usually do not concentrate on parallelization. Many approaches have been proposed to parallelize Python programs automatically, however, they are either domain-specific or require manual annotation. Thus they cannot solve the problem well in general. In this paper, we propose PyPar, an effective tool aiming at discovering parallelization possibilities in real-world Python programs. PyPar doesn’t need manual annotation and is universally applicable. It first drives a data-dependence analysis to determine whether two pieces of code can run concurrently. The key is the use of a graph-theoretic approach. Next, it adopts a dynamic selection strategy to eliminate inefficient parallelisms. Finally, PyPar produces a parallelism report as well as a referential parallelized program, which is built by PyPar using one of the three parallelization methods (thread-based, processbased, and Ray-based). We have implemented a prototype of PyPar and evaluated it on six well-designed widely-used real-world Python packages: Scikit-Image, SciPy, librosa, trimesh, Scikit-learn and seaborn. In total, 1,240 functions are tested, and PyPar found 127 parallelizable functions among them. Based on manual filtering, only 7 of them are false positives (i.e., a 94.5% precision). The remaining 120 are parallelizable (almost 10% among all functions under test), and most of them can be efficiently sped up by gaining an acceleration of up to 90% , with an average of 44%. The acceleration in practice is close to theoretical estimation. The results show that even well-designed practical Python programs can be further parallelized for speeding up, and PyPar can bring effective and efficient parallelization on real-world Python programs.