菜单

关于 🐙 GitHub
arXiv 提交日期: 2026-05-26
📄 Abstract - Learning When to Think While Listening in Large Audio-Language Models

Recent advances in Large Audio-Language Models (LALMs) have made real-time, streaming spoken interaction increasingly practical. In this setting, reasoning quality and responsiveness are tightly coupled: delaying reasoning until the speech endpoint can improve answer quality but moves deliberation into user-visible response delay, while answering too early risks committing before decisive evidence arrives. We introduce a learnable wait-think-answer control formulation for LALMs. Motivated by the incremental nature of human conversation, the controller decides under partial audio evidence when to wait, when to externalize a compact reasoning update, and when to answer. Using Qwen2.5-Omni-7B as the base model, we construct aligned wait-think-answer traces from spoken reasoning data, train the controller with supervised fine-tuning (SFT), and then apply Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO). The reward combines answer correctness, action validity, update timing, latency synchronization, reasoning quality, and chain consistency, optimizing the complete wait-think-answer trajectory and not the final answer alone. On a six-task synthetic spoken reasoning question answering (SRQA) benchmark, the six-reward DAPO controller improves the row-weighted accuracy from 67.6% to 70.3% while reducing post-endpoint final-think length by 14% under the same Qwen deployment harness. On a 186-item human-recorded Real Audio Bench, a transfer check beyond text-to-speech (TTS)-rendered speech, the controller family remains functional: SFT achieves the strongest accuracy, while the six-reward DAPO controller is the only learned variant whose final-think length falls below the base. These results suggest that a streaming model should learn when to make intermediate reasoning explicit during the audio stream.

顶级标签: audio llm reinforcement learning
详细标签: large audio-language models streaming reasoning wait-think-answer latency optimization policy optimization 或 搜索:

大型音频语言模型中“何时思考”的学习——在倾听中把握推理时机 / Learning When to Think While Listening in Large Audio-Language Models


1️⃣ 一句话总结

本文提出一种可学习的“等待-思考-回答”控制策略,让大型音频语言模型在实时语音交互中能自适应地决定何时继续倾听、何时进行中间推理、何时给出最终答案,从而在提升回答准确率的同时,有效缩短用户的等待延迟。

源自 arXiv: 2605.27190