一句话:提出建模驱动的采样方法 CoMSA,依据性能模型预测的不确定性挑选配置样本,提升性能建模精度并降低配置性能测试成本。
高度可配置系统允许用户在不同部署环境中灵活配置,但海量配置组合与有限的测试时间、资源之间的矛盾,使配置性能测试代价高昂。用性能模型近似预测配置性能是降低测试成本的主要手段,然而已有建模与采样技术在选择训练样本时通常不考虑性能模型自身的精度,导致实际建模效果次优。本文提出建模驱动的采样方法 CoMSA,其核心直觉是:若某配置经由当前性能模型得到的预测结果不确定性越高,就越应被选为后续训练样本,以此迭代改进模型。CoMSA 同时面向两类场景设计:没有历史性能测试结果的冷启动与已有历史结果的温启动。作者在 LRZIP、LLVM、x264 和 SQLite 四个对象上开展评估,结果表明相较其他基线采样方法,CoMSA 能显著提高预测模型的精度,并提升配置性能测试的效率。
Highly configurable systems enable customers to flexibly configure the systems in diverse deployment environments. The flexibility of configurations also poses challenges for performance testing. On one hand, there exist a massive number of possible configurations; while on the other hand, the time and resources are limited for performance testing, which is already a costly process during software development. Modeling the performance of configurations is one of the solutions to reduce the cost of configuration performance testing. Although prior research proposes various modeling and sampling techniques to build configuration performance models, the sampling approaches used in the model typically do not consider the accuracy of the performance models, leading to potential sub-optimal performance modeling results in practice. In this paper, we present a modeling-driven sampling approach (CoMSA) to improve the performance modeling of highly configurable systems. The intuition of CoMSA is to select samples based on their uncertainties to the performance models. In other words, the configurations that have the more uncertain performance prediction results by the performance models are more likely to be selected as further training samples to improve the model. CoMSA is designed by considering both scenarios where 1) the software projects do not have historical performance testing results (cold start) and 2) there exist historical performance testing results (warm start). We evaluate the performance of our approach in four subjects, namely LRZIP, LLVM, x264, and SQLite. Through the evaluation result, we can conclude that our sampling approaches could highly enhance the accuracy of the prediction models and the efficiency of configuration performance testing compared to other baseline sampling approaches.