GEM: Geometric Entropy Mixing for Optimal LLM Data Curation
The paper introduces GEM, a framework designed for optimal data curation in large language models (LLMs). It addresses issues in data mixing caused by flawed categorization and proposes a new approach using geometric entropy. Experimental results indicate that GEM improves downstream accuracy and establishes a new state-of-the-art in data mixing strategies.
- ▪GEM reformulates data curation as a variational problem on the hypersphere.
- ▪The framework employs a mixing-balance regularizer to counteract cluster collapse.
- ▪Experiments show that GEM improves average downstream accuracy by up to 1.2% when integrated into existing mixing strategies.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Machine Learning arXiv:2605.26121 (cs) [Submitted on 27 Apr 2026] Title:GEM: Geometric Entropy Mixing for Optimal LLM Data Curation Authors:Yue Min, Ziyun Qiao, Ruining Chen, Yujun Li View a PDF of the paper titled GEM: Geometric Entropy Mixing for Optimal LLM Data Curation, by Yue Min and 3 other authors View PDF HTML (experimental) Abstract:LLM pre-training efficacy increasingly depends on data composition rather than sheer volume. Yet, optimal mixing is hindered by categorization flaws: human taxonomies suffer from ontological misalignment, and Euclidean clustering fails to address embedding anisotropy.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
