一句话：提出基于非参数统计与自助法的云性能测试方法 PT4Cloud，用置信带和停止条件在降低测试成本的同时获得高精度性能分布。
云应用性能存在随机波动，公有云黑盒且使用成本高，使准确性能测试极其困难。本文提出云性能测试方法 PT4Cloud，利用非参数统计中的似然理论与 bootstrap 方法，提供可靠的停止条件，从而获得带置信带的高精度性能分布，并允许用户指定直观的精度目标，在精度与测试成本间权衡。作者在 AWS 和 Chameleon 云上用 33 个基准配置进行评估：与大量性能测试所得数据相比，PT4Cloud 平均准确率达 95.4%，同时减少 62% 的测试运行次数；其两种测试执行缩减技术可在保持平均 91% 准确率的情况下减少 90.1% 的运行次数。与三种其他技术相比，该方法结果更准确。
The low cost of resource ownership and flexibility have led users to increasingly port their applications to the clouds. To fully realize the cost benefits of cloud services, users usually need to reliably know the execution performance of their applications. However, due to the random performance fluctuations experienced by cloud applications, the black box nature of public clouds and the cloud usage costs, testing on clouds to acquire accurate performance results is extremely difficult. In this paper, we present a novel cloud performance testing methodology called PT4Cloud. By employing non-parametric statistical approaches of likelihood theory and the bootstrap method, PT4Cloud provides reliable stop conditions to obtain highly accurate performance distributions with confidence bands. These statistical approaches also allow users to specify intuitive accuracy goals and easily trade between accuracy and testing cost. We evaluated PT4Cloud with 33 benchmark configurations on Amazon Web Service and Chameleon clouds. When compared with performance data obtained from extensive performance tests, PT4Cloud provides testing results with 95.4% accuracy on average while reducing the number of test runs by 62%. We also propose two test execution reduction techniques for PT4Cloud, which can reduce the number of test runs by 90.1% while retaining an average accuracy of 91%. We compared our technique to three other techniques and found that our results are much more accurate.
DOI 原文 ·
@inproceedings{paperbot5173,
title = {A statistics-based performance testing methodology for cloud applications},
author = {Sen He and Glenna Manns and John Saunders and Wei Wang and Lori Pollock and Mary Lou Soffa},
booktitle = {FSE},
year = {2019},
doi = {10.1145/3338906.3338912}
}