视觉作为统一的多模态生成 / Vision as Unified Multimodal Generation
1️⃣ 一句话总结
本文提出将计算机视觉任务统一为多模态生成问题,通过一个通用模型(SenseNova-Vision)同时处理目标检测、图像分割、深度估计等多种视觉任务,无需特定任务的专用架构,仅需自然语言指令和可选的视觉提示,即可输出文本或图像结果,实验表明该统一模型在多项任务上可媲美专业系统。
We formulate computer vision as unified multimodal generation, where heterogeneous visual tasks are expressed in the native text and image generation spaces of a unified multimodal model, without task-specific architectures. Under this formulation, SenseNova-Vision uses natural-language instructions and optional visual prompts to specify tasks, target regions or views, and decoding conventions, and generates responses as text for symbolic outputs, images for dense spatial predictions, or mixed text-and-image outputs for compositional tasks. To support large-scale training, we convert diverse computer vision annotations into instruction-response examples compatible with these generation spaces, resulting in the SenseNova-Vision Corpus, a computer-vision instruction-response corpus spanning text, image, and mixed targets. Starting from an off-the-shelf pretrained unified multimodal model, SenseNova-Vision is trained primarily on this corpus, with auxiliary multimodal data used as a capability-preserving mixture, and requires no task-specific prediction heads or architectural modifications. The resulting model covers a broad range of vision tasks, including detection, OCR, keypoint estimation, segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation, while supporting language-defined variants that combine category, color, region, and other visual cues. Experiments show that a single unified model can match leading task-specialized systems across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. These results suggest unified multimodal generation as a scalable route for integrating computer vision capabilities into general-purpose foundation models. The model and corpus are publicly available.
视觉作为统一的多模态生成 / Vision as Unified Multimodal Generation
本文提出将计算机视觉任务统一为多模态生成问题,通过一个通用模型(SenseNova-Vision)同时处理目标检测、图像分割、深度估计等多种视觉任务,无需特定任务的专用架构,仅需自然语言指令和可选的视觉提示,即可输出文本或图像结果,实验表明该统一模型在多项任务上可媲美专业系统。
源自 arXiv: 2607.06560