一句话：提出 DPLOG，用统计采样为 Spark 应用链式数据处理步骤提供低开销日志、中间信息与诊断提示。
大数据处理框架（如 Apache Spark）的抽象让开发者专注于分析逻辑，却隐藏执行细节，使传统日志难以监控中间状态和诊断问题。本文先人工分析 Stack Overflow 上 1,000 个随机抽样的 Spark 问题，归纳根因及有助于监控诊断的信息类型；进而设计 DPLOG，利用统计采样降低开销，并为链式方法管线的每个数据处理步骤提供中间信息及提示/警告。在六个基准程序上，DPLOG 处理 5GB 数据时响应时间增加低于 10%，相比基线开销降低超过 500%；20 名开发者的用户研究显示调试时间减少 63%，有用性评分平均 4.85/5。该思路也可推广到其他大数据框架。
To analyze large-scale data efficiently, developers have created various big data processing frameworks (e.g., Apache Spark). These big data processing frameworks provide abstractions to developers so that they can focus on implementing the data analysis logic. In traditional software systems, developers leverage logging to monitor applications and record intermediate states to assist workload understanding and issue diagnosis. However, due to the abstraction and the peculiarity of big data frameworks, there is currently no effective monitoring approach for big data applications. In this paper, we first manually study 1,000 randomly sampled Spark-related questions on Stack Overflow to study their root causes and the type of information, if recorded, that can assist developers with motioning and diagnosis. Then, we design an approach, DPLOG, which assists developers with monitoring Spark applications. DPLOG leverages statistical sampling to minimize performance overhead and provides intermediate information and hint/warning messages for each data processing step of a chained method pipeline. We evaluate DPLOG on six benchmarking programs and find that DPLOG has a relatively small overhead (i.e., less than 10% increase in response time when processing 5GB data) compared to without using DPLOG, and reduce the overhead by over 500% compared to the baseline. Our user study with 20 developers shows that DPLOG can reduce the needed time to debug big data applications by 63% and the participants give DPLOG an average of 4.85/5 for its usefulness. The idea of DPLOG may be applied to other big data processing frameworks, and our study sheds light on future research opportunities in assisting developers with monitoring big data applications.
DOI 原文 ·
@inproceedings{paperbot6464,
title = {Would you like a quick peek? providing logging support to monitor data processing in big data applications},
author = {Zehao Wang and Haoxiang Zhang and Tse-Hsun (Peter) Chen and Shaowei Wang},
booktitle = {FSE},
year = {2021},
doi = {10.1145/3468264.3468613}
}