The Randomness Floor: Measuring Intrinsic Non-Randomness in Language Model Token Distributions
This study introduces Entropic Deviation (ED) to measure the intrinsic non-randomness in language model token distributions, revealing a structural 'randomness floor' across models and architectures. Transformers show consistent ED values around 0.30 under neutral prompts, indicating most non-randomness stems from learned parameters rather than context. Mamba2, a state space model, exhibits higher ED, lower variance, and greater temperature sensitivity compared to transformers. Cross-lingual tests show language-specific modulation of ED independent of tokenisation.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Computation and Language arXiv:2604.22771 (cs) [Submitted on 29 Mar 2026] Title:The Randomness Floor: Measuring Intrinsic Non-Randomness in Language Model Token Distributions Authors:Jarosław Hryszko View a PDF of the paper titled The Randomness Floor: Measuring Intrinsic Non-Randomness in Language Model Token Distributions, by Jaros{\l}aw Hryszko View PDF HTML (experimental) Abstract:Language models cannot be random. This paper introduces Entropic Deviation (ED), the normalised KL divergence between a model's token distribution and the uniform distribution, and measures it systematically across 31,200 generations spanning seven models, two architectures (transformer and state space), nine prompt categories, three temperatures, and five languages.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI