菜单

关于 🐙 GitHub
arXiv 提交日期: 2026-03-05
📄 Abstract - Wiki-R1: Incentivizing Multimodal Reasoning for Knowledge-based VQA via Data and Sampling Curriculum

Knowledge-Based Visual Question Answering (KB-VQA) requires models to answer questions about an image by integrating external knowledge, posing significant challenges due to noisy retrieval and the structured, encyclopedic nature of the knowledge base. These characteristics create a distributional gap from pretrained multimodal large language models (MLLMs), making effective reasoning and domain adaptation difficult in the post-training stage. In this work, we propose \textit{Wiki-R1}, a data-generation-based curriculum reinforcement learning framework that systematically incentivizes reasoning in MLLMs for KB-VQA. Wiki-R1 constructs a sequence of training distributions aligned with the model's evolving capability, bridging the gap from pretraining to the KB-VQA target distribution. We introduce \textit{controllable curriculum data generation}, which manipulates the retriever to produce samples at desired difficulty levels, and a \textit{curriculum sampling strategy} that selects informative samples likely to yield non-zero advantages during RL updates. Sample difficulty is estimated using observed rewards and propagated to unobserved samples to guide learning. Experiments on two KB-VQA benchmarks, Encyclopedic VQA and InfoSeek, demonstrate that Wiki-R1 achieves new state-of-the-art results, improving accuracy from 35.5\% to 37.1\% on Encyclopedic VQA and from 40.1\% to 44.1\% on InfoSeek. The project page is available at this https URL.

顶级标签: multi-modal model training natural language processing
详细标签: visual question answering knowledge-based reasoning curriculum learning reinforcement learning data generation 或 搜索:

Wiki-R1:通过数据和采样课程激励基于知识的视觉问答中的多模态推理 / Wiki-R1: Incentivizing Multimodal Reasoning for Knowledge-based VQA via Data and Sampling Curriculum


1️⃣ 一句话总结

这篇论文提出了一种名为Wiki-R1的课程学习框架,通过生成可控难度的训练数据和智能采样策略,帮助多模态大语言模型更好地结合外部知识来回答图片相关的问题,从而在两项视觉问答基准测试上取得了新的最佳性能。

源自 arXiv: 2603.05256