一句话：提出DAC，用层次建模与遗传算法对Spark等内存集群计算的高维配置进行数据规模感知自动调优。
内存集群计算框架（如Spark）通常重复运行相似输入规模的程序，但其性能对输入数据规模高度敏感，且性能关键配置参数超过40个，导致性能建模与调优困难。为此，论文提出DAC，一种数据规模感知的自动调优方法，将输入数据集大小与41个配置参数纳入给定IMC程序的性能模型；采用层次建模（HM）以层次方式组合多个子模型，并使用遗传算法（GA）搜索最优高维配置。评估使用六个典型Spark程序、每个五种输入数据规模，DAC相比默认配置平均加速30.4倍、最高89倍；相比默认、专家和RFHOC配置，几何平均加速分别为15.4倍、2.3倍和1.5倍。
In-Memory cluster Computing (IMC) frameworks (e.g., Spark) have become increasingly important because they typically achieve more than 10× speedups over the traditional On-Disk cluster Computing (ODC) frameworks for iterative and interactive applications. Like ODC, IMC frameworks typically run the same given programs repeatedly on a given cluster with similar input dataset size each time. It is challenging to build performance model for IMC program because: 1) the performance of IMC programs is more sensitive to the size of input dataset, which is known to be difficult to be incorporated into a performance model due to its complex effects on performance; 2) the number of performance-critical configuration parameters in IMC is much larger than ODC (more than 40 vs. around 10), the high dimensionality requires more sophisticated models to achieve high accuracy. To address this challenge, we propose DAC, a datasize-aware auto-tuning approach to efficiently identify the high dimensional configuration for a given IMC program to achieve optimal performance on a given cluster. DAC is a significant advance over the state-of-the-art because it can take the size of input dataset and 41 configuration parameters as the parameters of the performance model for a given IMC program, --- unprecedented in previous work. It is made possible by two key techniques: 1) Hierarchical Modeling (HM), which combines a number of individual sub-models in a hierarchical manner; 2) Genetic Algorithm (GA) is employed to search the optimal configuration. To evaluate DAC, we use six typical Spark programs, each with five different input dataset sizes. The evaluation results show that DAC improves the performance of six typical Spark programs, each with five different input dataset sizes compared to default configurations by a factor of 30.4x on average and up to 89x. We also report that the geometric mean speedups of DAC over configurations by default, expert, and RFHOC are 15.4x, 2.3x, and 1.5x, respectively.
DOI 原文 ·
@inproceedings{paperbot4274,
title = {Datasize-Aware High Dimensional Configurations Auto-Tuning of In-Memory Cluster Computing},
author = {Zhibin Yu and Zhendong Bei and Xuehai Qian},
booktitle = {ASPLOS},
year = {2018},
doi = {10.1145/3173162.3173187}
}