上下文模型预测生成:从语言模型到物理的开词汇运动合成 / In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics
1️⃣ 一句话总结
本文提出了一种名为ICMPG的新框架,通过将大语言模型的语义规划能力与物理模拟的实时反馈相结合,在从文本生成人类动作时,既保证了动作的语义准确性又确保了物理合理性,无需针对特定任务重新训练模型。
Synthesizing human motion from textual descriptions is essential for immersive digital applications, yet existing methods face a persistent trade-off between semantic fidelity and physical realism. Large language model (LLM)-based approaches can interpret diverse open-vocabulary instructions and compose high-level action plans, but they often generate motions that violate physical constraints. Physics-aware models improve realism through simulation or control, but they struggle with semantic complexity, fine-grained instructions, and novel concepts. To address this gap, we propose In-Context Model Predictive Generation (ICMPG), a framework that integrates language-model planning with inference-time physical feedback. ICMPG reformulates motion synthesis as a Model Predictive Control (MPC)-like process with two modules. The Context-Aware Motion Generation (CAMG) module uses an LLM as a planner to decompose textual commands and generate candidate motion sequences from motion tokens. The Model Predictive Generation (MPG) module evaluates these candidates through physical simulation and semantic alignment, estimates a composite reward, and selects the best sequence to guide subsequent generation steps. Unlike open-loop generation, this closed-loop refinement enables ICMPG to adapt motions to both the input semantics and the simulated physical environment without task-specific policy retraining. Extensive experiments across standard and zero-shot open-vocabulary settings show that ICMPG generalizes robustly to diverse commands and produces motions that are more physically plausible and semantically faithful than representative baselines on the evaluated benchmarks. The framework bridges semantic interpretation and physical simulation while remaining flexible enough to incorporate different LLM backbones, enabling more versatile and controllable text-driven motion synthesis.
上下文模型预测生成:从语言模型到物理的开词汇运动合成 / In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics
本文提出了一种名为ICMPG的新框架,通过将大语言模型的语义规划能力与物理模拟的实时反馈相结合,在从文本生成人类动作时,既保证了动作的语义准确性又确保了物理合理性,无需针对特定任务重新训练模型。
源自 arXiv: 2606.26981