一句话：比较三种用户行为粒度的工作负载恢复方法，发现带上下文的动作序列能生成更贴近现场且低成本的负载测试。
本文是一篇关于大规模系统负载测试中工作负载恢复的经验报告。粗粒度负载测试可能遗漏用户行为细节，而逐步回放用户操作又使测试的构建与维护成本过高。以往工作多以用户动作频率刻画负载，忽略了动作上下文与序列中的信息。作者提出三种分别基于基本用户动作、带上下文的基本用户动作、及带上下文的用户动作序列的恢复方法，并在四个系统（Apache James、OpenMRS、Google Borg 与阿里巴巴 SA）上评估。结果表明，基于带上下文动作序列的方法最优，其生成的负载测试在吞吐量与 CPU 使用上与原现场工作负载接近（多为统计不显著或小效应量），且仅需少量用户聚类即可生成，成本低；该方法还能以高精度与高召回率检测注入用户。
Designing field-representative load tests is an essential step for the quality assurance of large-scale systems. Practitioners may capture user behaviour at different levels of granularity. A coarse-grained load test may miss detailed user behaviour, leading to a non-representative load test; while an extremely fine-grained load test would simply replay user actions step by step, leading to load tests that are costly to develop, execute and maintain. Workload recovery is at core of these load tests. Prior research often captures the workload as the frequency of user actions. However, there exists much valuable information in the context and sequences of user actions. Such richer information would ensure that the load tests that leverage such workloads are more field-representative. In this experience paper, we study the use of different granularities of user behaviour, i.e., basic user actions, basic user actions with contextual information and user action sequences with contextual information, when recovering workloads for use in the load testing of large-scale systems. We propose three approaches that are based on the three granularities of user behaviour and evaluate our approaches on four subject systems, namely Apache James, OpenMRS, Google Borg, and an ultra-large-scale industrial system (SA) from Alibaba. Our results show that our approach that is based on user action sequences with contextual information outperforms the other two approaches and can generate more representative load tests with similar throughput and CPU usage to the original field workload (i.e., mostly statistically insignificant or with small/trivial effect sizes). Such representative load tests are generated only based on a small number of clusters of users, leading to a low cost of conducting/maintaining such tests. Finally, we demonstrate that our approaches can detect injected users in the original field workloads with high precision and recall. Our paper demonstrates the importance of user action sequences with contextual information in the workload recovery of large-scale systems.
DOI 原文 ·
@inproceedings{paperbot5273,
title = {An Experience Report of Generating Load Tests Using Log-Recovered Workloads at Varying Granularities of User Behaviour},
author = {Jinfu Chen and Weiyi Shang and Ahmed E. Hassan and Yong Wang and Jiangbin Lin},
booktitle = {ASE},
year = {2019},
doi = {10.1109/ase.2019.00068}
}