一句话：通过九系统大规模实证研究，揭示工作负载与配置项交互会显著影响性能模型泛化性。
现代可配置软件系统通常有大量配置项，其性能会随配置和工作负载变化。现有基于机器学习的性能预测模型大多只用单一工作负载测量，难以代表真实应用场景。本文对九个真实可配置软件系统、共25,258个配置开展系统性实证研究，考察工作负载单独及与配置项组合时对性能的影响，并利用配置项级代码覆盖率信息探索交互成因。结果表明，工作负载会造成显著性能变化，并与配置项以常常非单调的方式交互。这不仅削弱单一工作负载模型的泛化能力，也挑战现有迁移学习技术的假设。因此，构建性能预测模型时应纳入工作负载因素，以提高代表性和可靠性。
Modern software systems often exhibit numerous configuration options to tailor them to user requirements, including the system's performance behavior. Performance models derived via machine learning are an established approach for estimating and optimizing configuration-dependent software performance. Most existing approaches in this area rely on software performance measurements conducted with a single workload (i.e., input fed to a system). This single workload, however, is often not representative of a software system's real-world application scenarios. Understanding to what extent configuration and workload-individually and combined-cause a software system's performance to vary is key to understand whether performance models are generalizable across different configurations and workloads. Yet, so far, this aspect has not been systematically studied. To fill this gap, we conducted a systematic empirical study across 25 258 configurations from nine real-world configurable software systems to investigate the effects of workload variation at system-level performance and for individual configuration options. We explore driving causes for workload-configuration interactions by enriching performance observations with option-specific code coverage information. Our results demonstrate that workloads can induce substantial performance variation and interact with configuration options, often in non-monotonous ways. This limits not only the generalizability of single-workload models, but also challenges assumptions for existing transfer-learning techniques. As a result, workloads should be considered when building performance prediction models to maintain and improve representativeness and reliability.
@inproceedings{MuhlbauerSKDAS23,
title = {Analysing the Impact of Workloads on Modeling the Performance of Configurable Software Systems},
author = {Stefan Mühlbauer and Florian Sattler and Christian Kaltenecker and Johannes Dorn and Sven Apel and Norbert Siegmund},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00176}
}