超越预测准确率:用目标空间恢复概况评估模型与大脑的对齐程度 / Beyond Prediction Accuracy: Target-Space Recovery Profiles for Evaluating Model-Brain Alignment
1️⃣ 一句话总结
本文提出了一种新方法,不仅看模型预测大脑反应准不准,而是通过分析模型能否“恢复”大脑中可重复出现的反应特征,更精细地揭示模型与大脑之间的真正对齐程度,发现即使预测准确率相近,不同模型在恢复大脑不同维度特征时可能有本质差异。
Artificial vision models are often evaluated against the human visual cortex by measuring how accurately their internal representations predict brain responses. However, prediction accuracy alone does not indicate which dimensions of the target brain's response space are recovered. Here, we introduce a unified framework for evaluating both model-brain and brain-brain alignment by identifying the response dimensions recovered by prediction. Using repeated fMRI measurements, we first identify target-brain response dimensions that can be reproducibly predicted across independent trial splits. We then predict target-brain responses from either another subject's brain responses or a vision model's internal representations, and quantify how strongly each of these reproducible response dimensions is recovered. Applying this framework to a subset of the Natural Scenes Dataset, in which eight subjects viewed the same natural images during fMRI, we find that the early-to-intermediate visual-cortex responses contain a low-dimensional set of reproducible dimensions. Brain-to-brain comparisons identify which of these dimensions are consistently recoverable from other subjects' brains, providing a diagnostic human reference rather than only a scalar benchmark. In some cases, pretrained and randomly initialized models achieve similar prediction accuracy while showing distinct recovery profiles across these response dimensions. These results show that prediction accuracy alone can mask model-brain mismatches. By making explicit which reproducible brain response dimensions are recovered by prediction, our framework provides a more diagnostic evaluation of alignment between artificial vision models and the human visual cortex.
超越预测准确率:用目标空间恢复概况评估模型与大脑的对齐程度 / Beyond Prediction Accuracy: Target-Space Recovery Profiles for Evaluating Model-Brain Alignment
本文提出了一种新方法,不仅看模型预测大脑反应准不准,而是通过分析模型能否“恢复”大脑中可重复出现的反应特征,更精细地揭示模型与大脑之间的真正对齐程度,发现即使预测准确率相近,不同模型在恢复大脑不同维度特征时可能有本质差异。
源自 arXiv: 2605.20127