菜单

🤖 系统
📄 Abstract - Canvas-to-Image: Compositional Image Generation with Multimodal Controls

While modern diffusion models excel at generating high-quality and diverse images, they still struggle with high-fidelity compositional and multimodal control, particularly when users simultaneously specify text prompts, subject references, spatial arrangements, pose constraints, and layout annotations. We introduce Canvas-to-Image, a unified framework that consolidates these heterogeneous controls into a single canvas interface, enabling users to generate images that faithfully reflect their intent. Our key idea is to encode diverse control signals into a single composite canvas image that the model can directly interpret for integrated visual-spatial reasoning. We further curate a suite of multi-task datasets and propose a Multi-Task Canvas Training strategy that optimizes the diffusion model to jointly understand and integrate heterogeneous controls into text-to-image generation within a unified learning paradigm. This joint training enables Canvas-to-Image to reason across multiple control modalities rather than relying on task-specific heuristics, and it generalizes well to multi-control scenarios during inference. Extensive experiments show that Canvas-to-Image significantly outperforms state-of-the-art methods in identity preservation and control adherence across challenging benchmarks, including multi-person composition, pose-controlled composition, layout-constrained generation, and multi-control generation.

顶级标签: computer vision multi-modal model training
详细标签: image generation multimodal control diffusion models compositional generation spatial reasoning 或 搜索:

📄 论文总结

画布到图像:基于多模态控制的组合式图像生成 / Canvas-to-Image: Compositional Image Generation with Multimodal Controls


1️⃣ 一句话总结

这篇论文提出了一个名为Canvas-to-Image的统一框架,通过将文本、参考图像、空间布局等多种控制信号整合到一个画布中,并采用多任务联合训练,使AI模型能够更准确地生成符合用户复杂意图的组合图像。


📄 打开原文 PDF