一句话:提出 βMAX,一种无需格式规范、最大化数据恢复且对解析器约束更少的数据修复算法。
现代数据处理流水线依赖输入记录符合严格格式规范,但数据录入错误、传输损坏、格式不一致等会造成记录损坏,直接丢弃会损失信息。已有修复方法中,基于正则表达式或纠错解析器的方法需要格式规范,而 ddmax、eREPAIR 等无规范方法在修复操作、修复位置和解析器性质上受限。本文提出 βMAX:一种格式无关的数据修复算法,针对给定示例数据达到最优,并具有最大数据恢复和最少解析器约束。评估表明,βMAX 比 eREPAIR 所需信息更少,却能修复 83% 的损坏记录,达到 eREPAIR 的 1.77 倍,同时 oracle 调用减少 27.7 倍。
Modern data-processing pipelines rely on input records conforming to strict format specifications. In practice, however, data corruption can occur at numerous stages including data-entry error, corruption during input processing and retransmission, inconsistent formatting, and incompatible specifications. Such corrupted data can result in loss of records, reducing the accuracy of processing. Rather than discarding corrupted records and losing valuable information, one can attempt to repair the data. Data-repair solutions such as regular expression based repair and error-correcting parsers require a specification to perform structural repairs. Specification-free techniques such as ddmax and eREPAIR are limited in repair operations, repair location, and require specific parser properties that are often unavailable. To tackle this challenge, we introduce βMAX, a novel format-free data repair algorithm optimal with respect to the provided example data, with maximal data-recovery and minimal parser constraints. Despite requiring less information than eREPAIR, βMAX repairs 83% of all corrupt records—1.77× the rate achieved by eREPAIR—while using 27.7× fewer oracle calls.