一句话：提出 CacheCheck，通过动态分析执行轨迹自动检测 Spark 应用中 RDD 缓存相关的性能缺陷。
Apache Spark 使用弹性分布式数据集（RDD）抽象，开发者需通过 persist() 和 unpersist() 手动管理 RDD 缓存。错误的缓存决策会引发重复计算或浪费宝贵内存，导致严重的性能退化。本文提出 CacheCheck，自动检测 Spark 应用中的缓存相关缺陷。我们归纳了六种缓存相关缺陷模式，并通过分析 Spark 应用的执行轨迹进行动态检测。在六个真实 Spark 应用上的评估表明，CacheCheck 检测出 72 个此前未知的缓存缺陷，其中 28 个已被开发者修复。
Apache Spark has been widely used to build big data applications. Spark utilizes the abstraction of Resilient Distributed Dataset (RDD) to store and retrieve large-scale data. To reduce duplicate computation of an RDD, Spark can cache the RDD in memory and then reuse it later, thus improving performance. Spark relies on application developers to enforce caching decisions by using persist() and unpersist() APIs, e.g., which RDD is persisted and when the RDD is persisted / unpersisted. Incorrect RDD caching decisions can cause duplicate computations, or waste precious memory resource, thus introducing serious performance degradation in Spark applications. In this paper, we propose CacheCheck, to automatically detect cache-related bugs in Spark applications. We summarize six cache-related bug patterns in Spark applications, and then dynamically detect cache-related bugs by analyzing the execution traces of Spark applications. We evaluate CacheCheck on six real-world Spark applications. The experimental result shows that CacheCheck detects 72 previously unknown cache-related bugs, and 28 of them have been fixed by developers.
DOI 原文 ·
@inproceedings{paperbot6048,
title = {Detecting cache-related bugs in Spark applications},
author = {Hui Li and Dong Wang and Tianze Huang and Yu Gao and Wensheng Dou and Lijie Xu and Wei Wang and Jun Wei and Hua Zhong},
booktitle = {ISSTA},
year = {2020},
doi = {10.1145/3395363.3397353}
}