📄 论文总结
强化学习提升大语言模型对层级知识的遍历能力 / Reinforcement Learning Improves Traversal of Hierarchical Knowledge in LLMs
1️⃣ 一句话总结
这项研究发现强化学习并非像传统观点认为的那样会损害语言模型的记忆知识，而是通过提升模型在已有知识层级中搜索和导航的‘程序性技能’，使其在需要遍历结构化知识（如医疗代码）的回忆任务中表现更优。
🤖 系统
Abstract - Reinforcement Learning Improves Traversal of Hierarchical Knowledge in LLMs
Reinforcement learning (RL) is often credited with improving language model reasoning and generalization at the expense of degrading memorized knowledge. We challenge this narrative by observing that RL-enhanced models consistently outperform their base and supervised fine-tuned (SFT) counterparts on pure knowledge recall tasks, particularly those requiring traversal of hierarchical, structured knowledge (e.g., medical codes). We hypothesize these gains stem not from newly acquired data, but from improved procedural skills in navigating and searching existing knowledge hierarchies within the model parameters. To support this hypothesis, we show that structured prompting, which explicitly guides SFTed models through hierarchical traversal, recovers most of the performance gap (reducing 24pp to 7pp on MedConceptsQA for DeepSeek-V3/R1). We further find that while prompting improves final-answer accuracy, RL-enhanced models retain superior ability to recall correct procedural paths on deep-retrieval tasks. Finally our layer-wise internal activation analysis reveals that while factual representations (e.g., activations for the statement "code 57.95 refers to urinary infection") maintain high cosine similarity between SFT and RL models, query representations (e.g., "what is code 57.95") diverge noticeably, indicating that RL primarily transforms how models traverse knowledge rather than the knowledge representation itself.