一句话：提出程序合成方法 VRDSynth，用新 DSL 与合成算法从多语言视觉丰富文档中抽取实体关系，无需预训练数据即超越 LayoutXLM。
视觉丰富文档（如收据、病历、保险表单）中语义关联实体的抽取（例如为每个条目找到对应价格）是业务决策的关键，但现有模板方法难以适应新布局，深度学习方法则依赖大量预训练数据且效果有限。本文提出程序合成方法 VRDSynth，其两项创新是：一种新的领域特定语言（DSL），用于有效刻画文档实体间的空间与文本关系；以及一种合成算法，利用实体间频繁空间关系构造初始程序，通过等价归约剪枝搜索空间，并组合正例、负例与互斥程序以提升覆盖率。作者在 FUNSD 与 XFUND 两个主流 VRD 理解基准的语义实体链接任务上评估，涵盖 8 种语言的 1600 份表单。实验表明，VRDSynth 在无预训练数据的情况下，于 8 种语言中的 5 种超越预训练深度模型 LayoutXLM，在 FUNSD 上 F1 提升 42%，并在 7/8 语言上与 LayoutXLM 互补；同时存储与推理内存大幅降低（1M 与 380MB 对比 1.48GB 与 3GB），时间效率相近。
Businesses often need to query visually rich documents (VRDs), e.g., purchase receipts, medical records, and insurance forms, among many other forms from multiple vendors, to make informed decisions. As such, several techniques have been proposed to automatically extract independent entities of interest from VRDs such as extracting price tags from purchase receipts, etc. However, for extracting semantically linked entities, such as finding corresponding price tags for each item, these techniques either have limited capability in handling new layouts, e.g., template-based approaches, or require extensive amounts of pre-training data and do not perform well, e.g., deep-learning approaches. In this work, we introduce a program synthesis method, namely VRDSynth, to automatically generate programs to extract entity relations from multilingual VRDs. Two key novelties, which empower VRDSynth to tackle flexible layouts while requiring no pre-training data for extracting entity relations, include: (1) a new domain-specific language (DSL) to effectively capture the spatial and textual relations between document entities, and (2) a novel synthesis algorithm that makes use of frequent spatial relations between entities to construct initial programs, equivalent reduction to prune the search space, and a combination of positive, negative, and mutually exclusive programs to improve the coverage of programs. We evaluate our method on two popular VRD understanding benchmarks, namely FUNSD and XFUND, on the semantic entity linking task, consisting of 1,600 forms in 8 different languages. Experiments show that VRDSynth, despite having no prior pre-training data, outperforms the state-of-the-art pre-trained deep-learning approach, namely LayoutXLM, in 5 out of 8 languages. Noticeably, VRDSynth achieved an improvement of 42% over LayoutXLM in terms of F1 score on FUNSD while being complementary to LayoutXLM in 7/8 languages. Regarding efficiency, VRDSynth significantly improves the memory footprint required for storage and inference over LayoutXLM (1M and 380MB versus that of 1.48GB and 3GB required by LayoutXLM), while maintaining similar time efficiency despite the speed differences between the languages used for implementation (Python vs C++).
@inproceedings{paperbot8522,
title = {VRDSynth: Synthesizing Programs for Multilingual Visually Rich Document Information Extraction},
author = {Thanh-Dat Nguyen and Tung Do-Viet and Hung Nguyen-Duy and Tuan-Hai Luu and Hung Le and Bach Le and Patanamon Thongtanunam},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3680314}
}