一句话:用时间序列分类模型在运行时预测 Java 微基准预热阶段结束,从而更优地平衡性能测试结果质量与测试时间。
Java 软件由于即时编译(JIT)存在预热阶段,期间性能测量波动剧烈,会损害性能测试结果质量。为兼顾有效性与实用性,性能测试设计需在结果质量与测试时间之间取得合理折中。实践中通常采用固定次数的迭代预热(state-of-practice),研究界则提出在运行时动态停止预热迭代(state-of-the-art),但这些方法对预热结束的估计常偏少或偏多,导致结果质量下降或测试时间增加,社区对此仍缺乏共识。本文提出并研究一个基于 AI 的框架,在测试执行过程中动态终止预热迭代:利用时间序列分类(TSC)的最新进展,从测量片段中预测预热阶段何时结束。作者在 JMH 微基准执行得到的 50 万个测量片段上训练了三种不同的 TSC 模型。实验表明,该框架显著提升了现有实践与最新方法所给预热估计的准确度,并在最高 35.3% 的微基准上带来结果质量或测试时间的净改进,说明用 AI 动态估计预热结束可提升 Java 性能测试的成本效益。
Performance testing aims at uncovering efficiency issues of software systems. In order to be both effective and practical, the design of a performance test must achieve a reasonable trade-off between result quality and testing time. This becomes particularly challenging in Java context, where the software undergoes a warm-up phase of execution, due to just-in-time compilation. During this phase, performance measurements are subject to severe fluctuations, which may adversely affect quality of performance test results. Both practitioners and researchers have proposed approaches to mitigate this issue. Practitioners typically rely on a fixed number of iterated executions that are used to warm-up the software before starting to collect performance measurements (state-of-practice). Researchers have developed techniques that can dynamically stop warm-up iterations at runtime (state-of-the-art). However, these approaches often provide suboptimal estimates of the warm-up phase, resulting in either insufficient or excessive warm-up iterations, which may degrade result quality or increase testing time. There is still a lack of consensus on how to properly address this problem. Here, we propose and study an AI-based framework to dynamically halt warm-up iterations at runtime. Specifically, our framework leverages recent advances in AI for Time Series Classification (TSC) to predict the end of the warm-up phase during test execution. We conduct experiments by training three different TSC models on half a million of measurement segments obtained from JMH microbenchmark executions. We find that our framework significantly improves the accuracy of the warm-up estimates provided by state-of-practice and state-of-the-art methods. This higher estimation accuracy results in a net improvement in either result quality or testing time for up to +35.3% of the microbenchmarks. Our study highlights that integrating AI to dynamically estimate the end of the warm-up phase can enhance the cost-effectiveness of Java performance testing.