菜单

关于 🐙 GitHub
arXiv 提交日期: 2026-05-28
📄 Abstract - Native Audio-Visual Alignment for Generation

Joint audio-video generation aims to synthesize temporally synchronized and semantically coherent visual-acoustic content. However, existing open-source methods mainly rely on either dual-tower designs with posterior alignment or fully unified tri-modal designs that mix textual context, audio and video in one shared space. The former weakens fine-grained audio-video co-evolution, while the latter couples semantic conditioning with low-level synchronization. To address these limitations, we propose NAVA, a Native Audio-Visual Alignment framework for joint audio-video generation. NAVA is built upon context-conditioned native audio-visual alignment: it first establishes audio-video correspondence in a dedicated interaction space, and then uses external context to condition the joint denoising process. Specifically, NAVA is instantiated with an Align-then-Fuse MMDiT architecture, which transitions from modality-aware audio-video alignment to modality-shared joint denoising. Furthermore, we introduce Timbre-in-Context Conditioning to associate reference timbre cues with corresponding speech spans to achieve controllable speech timbre. Experiments on Verse-Bench and Seed-TTS, together with a user study, demonstrate that NAVA achieves superior video quality, precise audio-visual synchronization, competitive audio quality, and stronger reference-timbre controllability using only 6.3B parameters.

顶级标签: multi-modal aigc audio
详细标签: audio-visual generation joint generation controllable generation diffusion model synchronization 或 搜索:

面向生成的原生音视频对齐 / Native Audio-Visual Alignment for Generation


1️⃣ 一句话总结

本文提出了一种名为NAVA的音视频联合生成框架,通过先对齐音频与视频的对应关系、再结合外部文本条件引导生成过程的设计,有效解决了现有方法中音视频协同进化不足或语义条件与低级同步耦合的问题,仅用63亿参数即可生成高画质、同步精准、可控制音色的音视频内容。

源自 arXiv: 2605.30073