LLMind:面向视觉语言模型的仿生免训练自适应视觉表征方法 / LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Models
1️⃣ 一句话总结
这篇论文提出了一种名为LLMind的仿生免训练框架,它模仿人眼视觉的注视点编码和皮层放大机制,让视觉语言模型能用极少的像素高效地聚焦于图像的关键信息区域,从而在多种视觉问答任务上大幅提升性能并节省计算资源。
Vision-Language Models (VLMs) typically assume a uniform spatial fidelity across the entire field of view of visual inputs, dedicating equal precision to even the uninformative regions. By contrast, human vision is neither uniform nor static; it is adaptive, selective, and resource-efficient. In light of this, we present the first systematic analysis of bio-inspired visual representation methods, providing insights for more efficient and adaptive VLMs. We propose LLMind (Looking Like the Mind), a novel training-free framework that mimics foveated encoding and cortical magnification in human vision to achieve adaptive, efficient representations for VLMs under tight pixel budgets. Our key idea is to explore a Bio-inspired Adaptive Sampling Strategy (BASS), enabling a Mobius-parameterized module that performs non-uniform sampling while preserving global scene structure. On top of BASS, we introduce closed-loop semantic feedback (CSF) via test-time adaptation to align perceptual saliency with textual information from the frozen VLM. We evaluate LLMind against uniform and other sampling baselines across diverse scene-level and region-guided visual question answering benchmarks. The results show dramatic gains, with average improvements of +20% on VQAv2, +38% on Seed-Bench, and +37% on A-OKVQA compared to uniform sampling under tight pixel budgets. More surprisingly, LLMind retains up to 82%, 92%, and 97% of the full-resolution performance using only 1%, 3%, and 5% of the pixels, respectively. Moreover, LLMind is lightweight, plug-and-play, and compatible with existing VLMs without requiring architectural changes.
LLMind:面向视觉语言模型的仿生免训练自适应视觉表征方法 / LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Models
这篇论文提出了一种名为LLMind的仿生免训练框架,它模仿人眼视觉的注视点编码和皮层放大机制,让视觉语言模型能用极少的像素高效地聚焦于图像的关键信息区域,从而在多种视觉问答任务上大幅提升性能并节省计算资源。
源自 arXiv: 2603.14882