When LLMs exhibit uneven performance across planning tasks, these gaps are often attributed to task difficulty. We argue that this explanation is incomplete, as task-level variation may reflect distinct latent planning competencies rather than differences along a single ability spectrum. We study this question on ACPBench-Hard by evaluating multiple LLM families under varying test-time reasoning budgets and applying a multidimensional item response theory model to uncover the latent competency structure underlying LLM planning. The analysis reveals two principal dimensions that shape planning performance: operational reasoning, the ability to evaluate local action applicability and immediate state transitions, and structural enumeration, the ability to reason about goal reachability and landmark structure. Operational reasoning improving under model scaling and longer reasoning traces, while structural enumeration remains comparatively insensitive. Our findings motivate competency-level evaluation of LLM planning, shifting the focus from whether models improve overall to which planning competencies improve, under what conditions, and why.
当我们谈论大模型规划时我们在谈论什么：两种不同规划能力的证据 / What We Talk About When We Talk About LLM Planning: Evidence for Two Distinct Planning Abilities
本文通过分析大语言模型在不同规划任务中的表现差异，发现其规划能力并非单一维度，而是由两种独立能力构成——一种是处理具体操作步骤的‘操作推理’能力，另一种是理解整体目标与关键节点的‘结构枚举’能力，并且这两种能力对计算资源和模型规模的反应不同。
源自 arXiv: 2607.11197
