Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time
The paper introduces OP-Mix, a novel data mixing algorithm designed for language model training. It addresses the limitations of existing methods by providing a unified solution that operates throughout the entire training lifecycle. OP-Mix has demonstrated significant improvements in model performance while reducing computational costs.
- ▪Data mixing is crucial for combining different data sources during language model training.
- ▪OP-Mix improves average perplexity by 6.3% during pretraining compared to training without mixing.
- ▪The algorithm uses 66% and 95% less compute than traditional methods for continual learning.
arXiv cs.AI files mainly under ai research. We currently carry 1,128 of its stories.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Computation and Language arXiv:2605.15220 (cs) [Submitted on 13 May 2026] Title:Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time Authors:Michael Y. Hu, Apurva Gandhi, Kyunghyun Cho, Tal Linzen, Pratyusha Sharma View a PDF of the paper titled Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time, by Michael Y. Hu and Apurva Gandhi and Kyunghyun Cho and Tal Linzen and Pratyusha Sharma View PDF HTML (experimental) Abstract:Data mixing decides how to combine different sources or types of data and is a consequential problem throughout language model training. In pretraining, data composition is a key determinant of model quality; in continual learning and adaptation, it governs what is retained and acquired.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
