更少的数据，更快的训练：重复使用较小数据集通过采样偏差加速学习 / Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases
1️⃣ 一句话总结
本文发现，在训练过程中重复使用较小的数据集，反而能比使用更大的整体数据集更快地达到良好效果，这种加速源于小数据集带来的采样偏差促进了神经网络各层的均衡成长，尤其对逻辑推理类任务特别有效。
arXiv 提交日期: 2026-05-19
Abstract - Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases
This work investigates the ``small-vs-large gap'', where repeating on fewer samples can lead to compute saving during training compared to using a larger dataset. This is observed across algorithmic tasks, architectures and optimizers and cannot be explained using prior theory. We argue that the speedup comes from appropriate layer-wise growth enabled by sampling biases, which is more pronounced when the dataset size is smaller. We provide both theoretical analysis and empirical evidence from various interventions. Our results suggest that using a smaller dataset with more repetitions is not just a fallback strategy under data scarcity, but can be proactively leveraged as a favorable inductive biases for optimization, particularly in reasoning tasks.
源自 arXiv: 2605.20314