缓解大型视觉语言模型中的纠缠引导以降低幻觉 / Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction
1️⃣ 一句话总结
本文提出了一种名为MESA的即插即用框架,通过有选择性地干预模型内部信号来减少AI看图说话时产生的‘幻觉’(即文本与图像内容不符),同时避免了现有方法导致的输出变短或语言风格改变等问题。
Large Vision-Language Models (LVLMs) have achieved remarkable success across cross-modal tasks but remain hindered by hallucinations, producing textual outputs inconsistent with visual content. Existing methods mitigate hallucinations but often alter generation behavior, resulting in shorter outputs and shifted token distributions, especially in latent space steering approaches. We identify that this issue stems from entangled steering signals, where suppressing hallucinations inadvertently disrupts the model's intrinsic generation behavior. To address this, we propose MESA, an effective plug-and-play framework that performs controlled and selective latent intervention for hallucination mitigation. Specifically, MESA targets hallucination-relevant responses while preserving the model's original token distribution, enabling effective hallucination reduction without compromising generation behavior. Extensive experiments across diverse generative and discriminative benchmarks demonstrate that MESA consistently reduces hallucinations while better preserving generation behavior, outperforming prior methods across multiple LVLM families.
缓解大型视觉语言模型中的纠缠引导以降低幻觉 / Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction
本文提出了一种名为MESA的即插即用框架,通过有选择性地干预模型内部信号来减少AI看图说话时产生的‘幻觉’(即文本与图像内容不符),同时避免了现有方法导致的输出变短或语言风格改变等问题。
源自 arXiv: 2604.07914