菜单

关于 🐙 GitHub
arXiv 提交日期: 2025-12-16
📄 Abstract - OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value

The rapid evolution of Large Language Models (LLMs) is predicated on the quality and diversity of post-training datasets. However, a critical dichotomy persists: while models are rigorously benchmarked, the data fueling them remains a black box--characterized by opaque composition, uncertain provenance, and a lack of systematic evaluation. This opacity hinders reproducibility and obscures the causal link between data characteristics and model behaviors. To bridge this gap, we introduce OpenDataArena (ODA), a holistic and open platform designed to benchmark the intrinsic value of post-training data. ODA establishes a comprehensive ecosystem comprising four key pillars: (i) a unified training-evaluation pipeline that ensures fair, open comparisons across diverse models (e.g., Llama, Qwen) and domains; (ii) a multi-dimensional scoring framework that profiles data quality along tens of distinct axes; (iii) an interactive data lineage explorer to visualize dataset genealogy and dissect component sources; and (iv) a fully open-source toolkit for training, evaluation, and scoring to foster data research. Extensive experiments on ODA--covering over 120 training datasets across multiple domains on 22 benchmarks, validated by more than 600 training runs and 40 million processed data points--reveal non-trivial insights. Our analysis uncovers the inherent trade-offs between data complexity and task performance, identifies redundancy in popular benchmarks through lineage tracing, and maps the genealogical relationships across datasets. We release all results, tools, and configurations to democratize access to high-quality data evaluation. Rather than merely expanding a leaderboard, ODA envisions a shift from trial-and-error data curation to a principled science of Data-Centric AI, paving the way for rigorous studies on data mixing laws and the strategic composition of foundation models.

顶级标签: llm data benchmark
详细标签: data-centric ai post-training data dataset evaluation data lineage reproducibility 或 搜索:

OpenDataArena:一个用于评估模型训练后数据集价值的公平开放平台 / OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value


1️⃣ 一句话总结

这篇论文提出了一个名为OpenDataArena的开放平台,旨在解决大语言模型训练数据不透明的问题,通过建立一个包含统一训练评估流程、多维评分框架和数据溯源工具的生态系统,来系统性地衡量和比较不同训练数据集的内在价值,从而推动以数据为中心的AI研究。


源自 arXiv: 2512.14051