一句话：提出机器码级减速构造近似真值，用于评估 JIT 系统采样性能剖析器的准确性。
现代 JIT 运行时中，采样式性能剖析器的结果可能不准确且难以验证，因为复杂软硬件栈缺乏已知真值。本文提出一种通过机器码级减速来近似真值的方法：在不改变 Java 程序优化与编译决策及现代 CPU 执行行为的前提下，精确地按指定幅度减慢程序；并用硬件剖析验证主导基本块级减速准确。基于减速后的基准，作者确认 Async-profiler、JFR、JProfiler 与 YourKit 保持原有性能行为并按相同比例将运行时间归因到方法，同时发现缺失调试信息会导致定位错误。最后，通过减速特定基本块近似真值评估采样剖析器，发现不同剖析器准确性差异巨大。该工作为 JIT 系统提供了首个实用评估方法。
Optimizing performance on top of modern runtime systems with just-in-time (JIT) compilation is a challenge for a wide range of applications from browser-based applications on mobile devices to large-scale server applications. Developers often rely on sampling-based profilers to understand where their code spends its time. Unfortunately, sampling of JIT-compiled programs can give inaccurate and sometimes unreliable results. To assess accuracy of such profilers, we would ideally want to compare their results to a known ground truth. With the complexity of today's software and hardware stacks, such ground truth is unfortunately not available. Instead, we propose a novel technique to approximate a ground truth by accurately slowing down a Java program at the machine-code level, preserving its optimization and compilation decisions as well as its execution behavior on modern CPUs. Our experiments demonstrate that we can slow down benchmarks by a specific amount, which is a challenge because of the optimizations in modern CPUs, and we verified with hardware profiling that on a basic-block level, the slowdown is accurate for blocks that dominate the execution. With the benchmarks slowed down to specific speeds, we confirmed that Async-profiler, JFR, JProfiler, and YourKit maintain original performance behavior and assign the same percentage of run time to methods. Additionally, we identify cases of inaccuracy caused by missing debug information, which prevents the correct identification of the relevant source code. Finally, we tested the accuracy of sampling profilers by approximating the ground truth by the slowing down of specific basic blocks and found large differences in accuracy between the profilers. We believe, our slowdown-based approach is the first practical methodology to assess the accuracy of sampling profilers for JIT-compiling systems and will enable further work to improve the accuracy of profilers.
@article{paperbot2985,
title = {Divining Profiler Accuracy: An Approach to Approximate Profiler Accuracy through Machine Code-Level Slowdown},
author = {Humphrey Burchell and Stefan Marr},
journal = {Proceedings of the ACM on Programming Languages},
volume = {9},
number = {OOPSLA2},
year = {2025},
doi = {10.1145/3763180}
}