菜单

关于 🐙 GitHub
arXiv 提交日期: 2026-05-19
📄 Abstract - Structured Layout Priors for Robust Out-of-Distribution Visual Document Understanding

Vision-Language Models (VLMs) parse documents end-to-end but frequently break down on layouts unlike those seen in training. We attribute this to a two-hop bottleneck: before the decoder can extract content (Hop 2), it must first classify and localize the enclosing layout entity (Hop 1), and when the first hop fails the second collapses into omissions, malformed structure, or autoregressive repetition. We pre-resolve Hop 1 outside the decoder by running a lightweight RT-DETR detector, serializing its outputs in the parser's native DocTags vocabulary, and injecting them into the prompt alongside the full page image. Unlike analyze-then-parse approaches that crop the page, or prior prompt-level priors written in plain text, our prior shares the decoder's generation space and leaves the global image in view as a fallback when detections are noisy. On a 10k-page structural out-of-distribution benchmark, markdown F1 rises from $0.37$ to $0.92$; on the Chinese subset of OmniDocBench, table TEDS rises from $0.01$ to $0.36$; and on the 26k-page ViDoRe V3 benchmark, infinite-loop decoding failures drop across every industrial domain tested. These gains cost $15\%$ wall-clock latency and a median of $74$ prompt tokens, with no architectural change to the base VLM. An attention-level analysis further reveals a bimodal phase shift in which the decoder attends to injected layout tokens when emitting structure and to image patches when emitting content, consistent with the two-hop bottleneck being alleviated. Model weights will be released to support reproducibility.

顶级标签: multi-modal document understanding model evaluation
详细标签: visual document understanding out-of-distribution layout priors vision-language models benchmark 或 搜索:

用于鲁棒性分布外视觉文档理解的结构化布局先验 / Structured Layout Priors for Robust Out-of-Distribution Visual Document Understanding


1️⃣ 一句话总结

本文提出一种轻量级方法,通过在视觉语言模型解码前额外添加一个布局检测器,将识别到的文档结构以模型自身能理解的标签形式注入提示中,从而大幅提升模型对从未见过的新颖页面布局的理解能力,解决了传统模型在未知布局上容易出错或陷入无限循环的问题。

源自 arXiv: 2605.19866