一句话：提出 Denum，通过数值 token 解析与字符串处理结合通用压缩器，实现高压缩比、高速度的日志压缩。
现有基于解析器的日志压缩器先按解析结果分组再交给通用压缩器，但解析与压缩目标不一致、性能依赖样本日志且处理慢。本文提出 Denum：利用日志中多数 token 为数值 token 的洞察，设计数值 token 解析模块，提取纯数字、含特殊字符数字及数值变量，并采用差分等定制处理（如时间戳差值）；字符串处理模块处理剩余无数字内容；两模块输出再交由通用压缩器。在 16 个日志数据集上，平均压缩比提升 8.7%–434.7%，平均压缩速度提升 2.6×–37.7×（26.2 MB/s）；将数值 token 解析模块集成到现有压缩器可提升平均压缩比 11.8%、速度约 37%。
Parser-based log compressors have been widely explored in recent years because the explosive growth of log volumes makes the compression performance of general-purpose compressors unsatisfactory. These parser-based compressors preprocess logs by grouping the logs based on the parsing result and then feed the preprocessed files into a general-purpose compressor. However, parser-based compressors have their limitations. First, the goals of parsing and compression are misaligned, so the inherent characteristics of logs were not fully utilized. In addition, the performance of parser-based compressors depends on the sample logs and thus it is very unstable. Moreover, parser-based compressors often incur a long processing time. To address these limitations, we propose Denum, a simple, general log compressor with high compression ratio and speed. The core insight is that a majority of the tokens in logs are numeric tokens (i.e. pure numbers, tokens with only numbers and special characters, and numeric variables) and effective compression of them is critical for log compression. Specifically, Denum contains a Numeric Token Parsing module, which extracts all numeric tokens and applies tailored processing methods (e.g. store the differences of incremental numbers like timestamps), and a String Processing module, which processes the remaining log content without numbers. The processed files of the two modules are then fed as input to a general-purpose compressor and it outputs the final compression results. Denum has been evaluated on 16 log datasets and it achieves an 8.7% -- 434.7% higher average compression ratio and 2.6× -- 37.7× faster average compression speed (i.e. 26.2 MB/S) compared to the baselines. Moreover, integrating Denum's Numeric Token Parsing module into existing log compressors can provide a 11.8% improvement in their average compression ratio and achieve 37% faster average compression speed.
DOI 原文 ·
@inproceedings{paperbot8296,
title = {Unlocking the Power of Numbers: Log Compression via Numeric Token Parsing},
author = {Siyu Yu and Yifan Wu and Ying Li and Pinjia He},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695474}
}