一句话：针对虚拟机负载缺乏稳态的现象，提出分段稳态的性能评估方法，并显著降低置信区间失效频率。
近期虚拟机预热研究表明，即使在严格控制下，小型确定性微基准也常无法达到峰值性能的稳态，这会破坏以预热后性能为中心的评估方法所依赖的平均性能和置信区间假设。本文在相对更大的虚拟机负载中考察这一现象，记录并分析类似的稳态缺失，认为它是此类负载的固有属性而非缺陷。作者提出一种更新后的性能评估方法，面向执行中呈现若干稳态性能片段、并由突然性能变化分隔的负载。基于 Java 虚拟机 Renaissance 基准套件，实验显示新方法生成的置信区间漏掉真实性能的频率比现有方法低 20% 以上。
Recent studies of virtual machine warm up have pointed out that even small deterministic microbenchmarks executed in tightly controlled circumstances often do not reach a steady state of peak performance. This impacts performance evaluation methodologies that focus on performance after warm up, because the lack of a steady state may violate common assumptions made when computing metrics such as the average performance or the confidence interval for that average. Our work examines the reported lack of steady state in the context of comparatively larger virtual machine workloads. We document and analyze similar lack of steady state and argue that it should be considered an inherent property of these workloads rather than a fault. We introduce an updated performance evaluation methodology for workloads whose execution exhibits segments of steady state performance separated by sudden performance changes. Using the Renaissance benchmark suite for the Java Virtual Machine, we show that the methodology can produce confidence intervals that miss the true performance over 20 % less often than the existing methodologies.
@article{paperbot12078,
title = {Experimental Evaluation Methodology for the Era of No Steady Performance},
author = {Jaromír Antoch and Walter Binder and Lubomír Bulej and François Farquet and Vojtěch Horký and Aleksandar Prokopec and Andrea Rosà and Petr Tůma},
journal = {Proceedings of the ACM on Programming Languages},
volume = {10},
number = {OOPSLA2},
year = {2026},
doi = {10.1145/3839448}
}