一句话：提出 Log3C，利用日志序列聚类与 KPI 关联，快速精准识别影响系统 KPI 的服务问题。
针对云在线系统日志规模巨大且高度不平衡、仅少量日志揭示影响 KPI 的问题、问题类型多样难以人工区分，本文提出基于聚类的 Log3C 方法。它同时利用日志序列（日志事件序列）与系统 KPI，设计级联聚类算法，通过迭代采样、聚类和匹配日志序列，在保持高准确率的同时显著节省聚类时间；再将日志序列簇与系统 KPI 关联以识别有影响力的问题。在 Microsoft 在线服务系统真实日志上评估，结果验证了有效性与效率，并已在工业实践中成功应用。
Logs are often used for troubleshooting in large-scale software systems. For a cloud-based online system that provides 24/7 service, a huge number of logs could be generated every day. However, these logs are highly imbalanced in general, because most logs indicate normal system operations, and only a small percentage of logs reveal impactful problems. Problems that lead to the decline of system KPIs (Key Performance Indicators) are impactful and should be fixed by engineers with a high priority. Furthermore, there are various types of system problems, which are hard to be distinguished manually. In this paper, we propose Log3C, a novel clustering-based approach to promptly and precisely identify impactful system problems, by utilizing both log sequences (a sequence of log events) and system KPIs. More specifically, we design a novel cascading clustering algorithm, which can greatly save the clustering time while keeping high accuracy by iteratively sampling, clustering, and matching log sequences. We then identify the impactful problems by correlating the clusters of log sequences with system KPIs. Log3C is evaluated on real-world log data collected from an online service system at Microsoft, and the results confirm its effectiveness and efficiency. Furthermore, our approach has been successfully applied in industrial practice.
DOI 原文 ·
@inproceedings{paperbot4599,
title = {Identifying impactful service system problems via log analysis},
author = {Shilin He and Qingwei Lin and Jian-Guang Lou and Hongyu Zhang and Michael R. Lyu and Dongmei Zhang},
booktitle = {FSE},
year = {2018},
doi = {10.1145/3236024.3236083}
}