一句话：提出 Logzip，利用迭代聚类提取日志隐藏结构并生成中间表示，显著提升日志压缩率。
系统日志记录软件运行时信息，是大规模软件工程任务的重要数据源，但长期归档消耗大量存储和计算资源，传统通用压缩工具如 gzip 对日志效果有限。本文提出 Logzip，一种面向系统日志的高效压缩方法。它通过快速迭代聚类从原始日志中提取隐藏结构，并生成连贯的中间表示，使后续压缩更有效。作者在五个不同系统类型的日志数据集上评估，总大小 63.6 GB。结果表明，Logzip 平均比传统压缩工具节省约一半存储空间，同时设计高度并行、额外开销可忽略。论文还分享了将 Logzip 应用于华为真实产品的工业经验。
System logs record detailed runtime information of software systems and are used as the main data source for many tasks around software engineering. As modern software systems are evolving into large scale and complex structures, logs have become one type of fast-growing big data in industry. In particular, such logs often need to be stored for a long time in practice (e.g., a year), in order to analyze recurrent problems or track security issues. However, archiving logs consumes a large amount of storage space and computing resources, which in turn incurs high operational cost. Data compression is essential to reduce the cost of log storage. Traditional compression tools (e.g., gzip) work well for general texts, but are not tailed for system logs. In this paper, we propose a novel and effective log compression method, namely logzip. Logzip is capable of extracting hidden structures from raw logs via fast iterative clustering and further generating coherent intermediate representations that allow for more effective compression. We evaluate logzip on five large log datasets of different system types, with a total of 63.6 GB in size. The results show that logzip can save about half of the storage space on average over traditional compression tools. Meanwhile, the design of logzip is highly parallel and only incurs negligible overhead. In addition, we share our industrial experience of applying logzip to Huawei's real products.
@inproceedings{paperbot5189,
title = {Logzip: Extracting Hidden Structures via Iterative Clustering for Log Compression},
author = {Jinyang Liu and Jieming Zhu and Shilin He and Pinjia He and Zibin Zheng and Michael R. Lyu},
booktitle = {ASE},
year = {2019},
doi = {10.1109/ase.2019.00085}
}