Information-Theoretic Measures in AI: A Practical Decision Guide
Information-theoretic (IT) measures are ubiquitous in artificial intelligence: entropy drives decision-tree splits and uncertainty quantification, cross-entropy is the default classification loss, mutual information underpins representation learning and feature selection, and transfer entropy reveals directed influence in dynamical systems. A second, less consolidated family of measures, integrated information (Phi), effective information (EI), and autonomy, has emerged for characterizing agent complexity. Despite wide adoption, measure selection is often decoupled from estimator assumptions, failure modes, and safe inferential claims. This paper provides a practical decision framework for all seven measures, organized around three prescriptive questions for each: (i) what question does the measure answer and in which AI context; (ii) which estimator is appropriate for the data type and dimensionality; and (iii) what is the most dangerous misuse. The framework is operationalized in two complementary artifacts: a measure-selection flowchart and a master decision table. We cover both AI/ML and decision-making agent application domains per measure, with standardized Bridge Boxes linking IT quantities to cognitive constructs. Three worked examples illustrate the framework on concrete practitioner scenarios spanning representation learning, temporal influence analysis, and evolved agent complexity.
Full article excerpt tap to expand
Computer Science > Artificial Intelligence arXiv:2604.23716 (cs) [Submitted on 26 Apr 2026] Title:Information-Theoretic Measures in AI: A Practical Decision Guide Authors:Nikolaos Al.Papadopoulos, Konstantinos E. Psannis View a PDF of the paper titled Information-Theoretic Measures in AI: A Practical Decision Guide, by Nikolaos Al.Papadopoulos and Konstantinos E. Psannis View PDF HTML (experimental) Abstract:Information-theoretic (IT) measures are ubiquitous in artificial intelligence: entropy drives decision-tree splits and uncertainty quantification, cross-entropy is the default classification loss, mutual information underpins representation learning and feature selection, and transfer entropy reveals directed influence in dynamical systems. A second, less consolidated family of measures, integrated information (Phi), effective information (EI), and autonomy, has emerged for characterizing agent complexity. Despite wide adoption, measure selection is often decoupled from estimator assumptions, failure modes, and safe inferential claims. This paper provides a practical decision framework for all seven measures, organized around three prescriptive questions for each: (i) what question does the measure answer and in which AI context; (ii) which estimator is appropriate for the data type and dimensionality; and (iii) what is the most dangerous misuse. The framework is operationalized in two complementary artifacts: a measure-selection flowchart and a master decision table. We cover both AI/ML and decision-making agent application domains per measure, with standardized Bridge Boxes linking IT quantities to cognitive constructs. Three worked examples illustrate the framework on concrete practitioner scenarios spanning representation learning, temporal influence analysis, and evolved agent complexity. Comments: 25 pages, 2 tables, 1 figure. Submitted to Applied Intelligence (Springer) Subjects: Artificial Intelligence (cs.AI); Information Theory (cs.IT); Machine Learning (cs.LG); Multiagent Systems (cs.MA) MSC classes: 94A17, 68T05 ACM classes: H.1.1; I.2.0; I.2.6 Cite as: arXiv:2604.23716 [cs.AI] (or arXiv:2604.23716v1 [cs.AI] for this version) https://doi.org/10.48550/arXiv.2604.23716 Focus to learn more arXiv-issued DOI via DataCite (pending registration) Submission history From: Nikolaos Papadopoulos [view email] [v1] Sun, 26 Apr 2026 14:00:22 UTC (64 KB) Full-text links: Access Paper: View a PDF of the paper titled Information-Theoretic Measures in AI: A Practical Decision Guide, by Nikolaos Al.Papadopoulos and Konstantinos E. PsannisView PDFHTML (experimental)TeX Source view license Current browse context: cs.AI < prev | next > new | recent | 2026-04 Change to browse by: cs cs.IT cs.LG cs.MA math math.IT References & Citations NASA ADSGoogle Scholar Semantic Scholar export BibTeX citation Loading... BibTeX formatted citation × loading... Data provided by: Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv (What is alphaXiv?) Links to Code Toggle CatalyzeX Code Finder for Papers (What is CatalyzeX?) DagsHub Toggle DagsHub (What is DagsHub?) GotitPub Toggle Gotit.pub (What is…
This excerpt is published under fair use for community discussion. Read the full article at arXiv.org.
Discussion
0 commentsMore from arXiv.org