Features have life history. And we should care
The paper discusses the life history of features in language models, highlighting their emergence, persistence, and eventual decline during training. It identifies a stable representational backbone that organizes the model's structure and outlines its four key properties. The findings suggest that the initial phase of training is crucial for establishing this scaffold, which influences the model's development throughout the training process.
- ▪Features in language models have a life history that includes emergence, persistence, and decline during training.
- ▪The study identifies approximately 50 sparse features that serve as a stable scaffold for the model's representational structure.
- ▪The first 1% of training is critical for assembling features, which reorganize significantly faster during this period.
arXiv cs.AI files mainly under ai research. We currently carry 1,128 of its stories.
Opening excerpt (first ~120 words) tap to expand
Quantitative Biology > Neurons and Cognition arXiv:2605.18789 (q-bio) [Submitted on 7 May 2026] Title:Features have life history. And we should care Authors:Philipp Stecher, Sandro Radovanović, Vlasta Sikimić, Reinhard Kahle View a PDF of the paper titled Features have life history. And we should care, by Philipp Stecher and Sandro Radovanovi\'c and Vlasta Sikimi\'c and Reinhard Kahle View PDF HTML (experimental) Abstract:Features in language models have life history: they emerge, persist, and die during training, yet the importance of that history remains largely unexplored.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
