一句话：提出差分性能分析的模糊测试与决策树解释方法，发现并定位机器学习库中的性能缺陷。
性能退化类编程错误广泛存在，但缺乏有效的发现与诊断工具。本文提出差分性能分析方法：寻找规模相同但性能差异巨大的输入，并以按规模参数化的输入类为单位比较性能函数，从而保证差异在大输入上依然稳健。方法分为两阶段——基于进化模糊测试的发现与基于决策树分类器的解释，二者均借助聚类提升效率；解释结果以可直接用于修复性能缺陷的形式给出程序输入与内部信息（如方法和条件）。作者在微基准上证明该方法优于现有模糊器，并在真实机器学习库中发现并解释多个性能缺陷，例如 scikit-learn 中逻辑回归的实现；其中四个缺陷系本文首次报告，并已被开发者修复。
Programming errors that degrade the performance of systems are widespread, yet there is very little tool support for finding and diagnosing these bugs. We present a method and a tool based on differential performance analysis---we find inputs for which the performance varies widely, despite having the same size. To ensure that the differences in the performance are robust (i.e. hold also for large inputs), we compare the performance of not only single inputs, but of classes of inputs, where each class has similar inputs parameterized by their size. Thus, each class is represented by a performance function from the input size to performance. Importantly, we also provide an explanation for why the performance differs in a form that can be readily used to fix a performance bug. The two main phases in our method are discovery with fuzzing and explanation with decision tree classifiers, each of which is supported by clustering. First, we propose an evolutionary fuzzing algorithm to generate inputs that characterize different performance functions. For this fuzzing task, the unique challenge is that we not only need the input class with the worst performance, but rather a set of classes exhibiting differential performance. We use clustering to merge similar input classes which significantly improves the efficiency of our fuzzer. Second, we explain the differential performance in terms of program inputs and internals (e.g., methods and conditions). We adapt discriminant learning approaches with clustering and decision trees to localize suspicious code regions. We applied our techniques on a set of micro-benchmarks and real-world machine learning libraries. On a set of micro-benchmarks, we show that our approach outperforms state-of-the-art fuzzers in finding inputs to characterize differential performance. On a set of case-studies, we discover and explain multiple performance bugs in popular machine learning frameworks, for instance in implementations of logistic regression in scikit-learn. Four of these bugs, reported first in this paper, have since been fixed by the developers.
@inproceedings{paperbot6073,
title = {Detecting and understanding real-world differential performance bugs in machine learning libraries},
author = {Saeid Tizpaz-Niari and Pavol Černý and Ashutosh Trivedi},
booktitle = {ISSTA},
year = {2020},
doi = {10.1145/3395363.3404540}
}