基于相似度引导的课程微调:用于神经架构合成的大型语言模型 / Similarity-Guided Curriculum Fine-Tuning of LLMs for Neural Architecture Synthesis
1️⃣ 一句话总结
本文提出了一种新方法,通过让大型语言模型先从简单的神经网络架构代码示例开始学习,再逐步接触更复杂的例子,从而更有效地自动设计神经网络结构,实验表明这种方法在高相似度样本上表现优异,但需要结合代码修复技术来应对权重漂移带来的性能下降。
Introduce a MinHash-based similarity scheduling framework that constructs a progressive curriculum over neural architecture code for LLM-based neural architecture search (NAS). Using 128-permutation MinHash signatures over normalised 7-gram source code shingles, we partition the reference pool into similarity bands and present them in increasing architectural heterogeneity, with the best LoRA adapter from each stage merged cumulatively into the backbone. We evaluate the framework on OlympicCoder-7B within the LEMUR benchmark on CIFAR-10 image classification, generating N =15 candidate architectures per epoch across six progressive fine-tuning steps. The curriculum achieves 60% peak success rate at the high-similarity level without post-processing repair. A 2*2 ablation at the most diverse level curriculum versus base model, with versus without partial interface repair reveals that without repair the base model (47% peak SR) substantially outperforms the curriculum model (7% SR), while adding partial repair brings both to 53% SR. This pattern is consistent with merge-level weight drift progressively erasing evaluator-interface priors, and suggests that interface repair and curriculum scheduling target distinct failure modes. We further report a cross-dataset transfer observation on SVHN, where direct base-model generation without curriculum warmup yields 27% peak SR at substantially lower accuracy (60.5%) than the CIFAR-10 equivalent, consistent with the increased synthesis difficulty of the unq-family anchor architecture.
基于相似度引导的课程微调:用于神经架构合成的大型语言模型 / Similarity-Guided Curriculum Fine-Tuning of LLMs for Neural Architecture Synthesis
本文提出了一种新方法,通过让大型语言模型先从简单的神经网络架构代码示例开始学习,再逐步接触更复杂的例子,从而更有效地自动设计神经网络结构,实验表明这种方法在高相似度样本上表现优异,但需要结合代码修复技术来应对权重漂移带来的性能下降。
源自 arXiv: 2607.11591