一句话：提出 LogShrink，通过分析日志消息的共性与变异性，显著提升日志压缩比并保持合理压缩速度。
随着系统规模扩大，日志数据量爆炸式增长，给日志存储带来巨大开销。现有通用和日志专用压缩方法未能充分利用日志数据特征。本文通过实证研究总结出三条有利于日志压缩的日志数据特征，并据此提出 LogShrink。该方法设计基于最长公共子序列和熵技术的分析器，识别日志消息中潜在的共性与变异性，从而以更短表示压缩日志；同时引入基于聚类的序列采样器加速分析。大量实验表明，LogShrink 在压缩比上平均优于基线 16% 至 356%，同时保持合理的压缩速度。
Log data is a crucial resource for recording system events and states during system execution. However, as systems grow in scale, log data generation has become increasingly explosive, leading to an expensive overhead on log storage, such as several petabytes per day in production. To address this issue, log compression has become a crucial task in reducing disk storage while allowing for further log analysis. Unfortunately, existing general-purpose and log-specific compression methods have been limited in their ability to utilize log data characteristics. To overcome these limitations, we conduct an empirical study and obtain three major observations on the characteristics of log data that can facilitate the log compression task. Based on these observations, we propose LogShrink, a novel and effective log compression method by leveraging commonality and variability of log data. An analyzer based on longest common subsequence and entropy techniques is proposed to identify the latent commonality and variability in log messages. The key idea behind this is that the commonality and variability can be exploited to shrink log data with a shorter representation. Besides, a clustering-based sequence sampler is introduced to accelerate the commonality and variability analyzer. The extensive experimental results demonstrate that LogShrink can exceed baselines in compression ratio by 16% to 356% on average while preserving a reasonable compression speed.
@inproceedings{paperbot11418,
title = {LogShrink: Effective Log Compression by Leveraging Commonality and Variability of Log Data},
author = {Xiaoyun Li and Hongyu Zhang and Van-Hoang Le and Pengfei Chen},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3608129}
}