paperbot · PL 论文追踪

全部论文 待读 收藏

P2 Discovering Parallelisms in Python Programs

发现 Python 程序中的并行性
FSE Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering2023匹配度 2/10 · 该论文研究 Python 程序的自动并行化发现,核心是数据依赖分析与动态筛选,属于传统程序分析方向,但并未涉及 LLM/Agent、浮点程序分析或贝叶斯程序分析,也没有面向 ML 系统或 Agentic 程序的静态分析,因此与用户当前核心研究方向匹配度较低。引用 1 inbox
Siwei Wei, Guyang Song, Senlin Zhu, Ruoyi Ruan, Shihao Zhu, Yan Cai
crossref: 2026-09-12dblp: 2026-09-12

AI 速览

一句话:提出 PyPar,自动发现真实 Python 程序的并行化机会,并生成并行化报告与参考并行实现。

Python 程序常存在未被开发者利用的并行化机会,现有自动并行化方法多局限于特定领域或需手工标注,难以通用。本文提出 PyPar:先通过数据依赖分析判断代码能否并发执行,核心是图论方法;再用动态筛选策略剔除低效并行;最后输出并行性报告,并基于线程、进程或 Ray 生成参考并行化程序。作者实现原型并在 Scikit-Image、SciPy、librosa、trimesh、Scikit-learn 和 seaborn 六个真实包上评估,共测试 1,240 个函数,发现 127 个可并行函数;人工过滤后仅 7 个误报,精度 94.5%。120 个可并行函数约占测试函数 10%,多数可获最高 90%、平均 44% 的加速,接近理论估计。

问题
开发者常忽略 Python 程序中的并行化机会,现有自动并行化方法或领域特定或需手工标注,通用性不足。
方法
PyPar 使用图论数据依赖分析判断并发可行性,并用动态选择策略过滤低效并行,最终生成并行性报告与基于线程、进程或 Ray 的参考并行程序。
评估
在六个广泛使用的真实 Python 包上测试 1,240 个函数,发现 127 个可并行函数,精度 94.5%,多数获得平均 44%、最高 90% 的加速。
原文摘要(Abstract)

Parallelization is a promising way to improve the performance of Python programs. Unfortunately, developers may miss parallelization possibilities, because they usually do not concentrate on parallelization. Many approaches have been proposed to parallelize Python programs automatically, however, they are either domain-specific or require manual annotation. Thus they cannot solve the problem well in general. In this paper, we propose PyPar, an effective tool aiming at discovering parallelization possibilities in real-world Python programs. PyPar doesn’t need manual annotation and is universally applicable. It first drives a data-dependence analysis to determine whether two pieces of code can run concurrently. The key is the use of a graph-theoretic approach. Next, it adopts a dynamic selection strategy to eliminate inefficient parallelisms. Finally, PyPar produces a parallelism report as well as a referential parallelized program, which is built by PyPar using one of the three parallelization methods (thread-based, processbased, and Ray-based). We have implemented a prototype of PyPar and evaluated it on six well-designed widely-used real-world Python packages: Scikit-Image, SciPy, librosa, trimesh, Scikit-learn and seaborn. In total, 1,240 functions are tested, and PyPar found 127 parallelizable functions among them. Based on manual filtering, only 7 of them are false positives (i.e., a 94.5% precision). The remaining 120 are parallelizable (almost 10% among all functions under test), and most of them can be efficiently sped up by gaining an acceleration of up to 90% , with an average of 44%. The acceleration in practice is close to theoretical estimation. The results show that even well-designed practical Python programs can be further parallelized for speeding up, and PyPar can bring effective and efficient parallelization on real-world Python programs.

链接与引用

DOI 原文 · PDF(开放获取) · DBLP

BibTeX
@inproceedings{WeiSZRZ023,
  title = {Discovering Parallelisms in Python Programs},
  author = {Siwei Wei and Guyang Song and Senlin Zhu and Ruoyi Ruan and Shihao Zhu and Yan Cai},
  booktitle = {FSE},
  year = {2023},
  doi = {10.1145/3611643.3616259}
}