1 Introduction
1.1 Why entropy?
Modern machine learning has long been nurtured by scientific disciplines. A canonical example is the Boltzmann machine, rooted in statistical physics and motivated by the Maximum Entropy Principle. This book takes entropy — and its close relative, energy — as the organizing thread for a family of learning methods that are both principled and practically effective.
That lineage is unusually direct. In 1982 John Hopfield borrowed the energy function of a magnetic spin system to build an associative memory (Hopfield 1982); within a year, simulated annealing had turned the competition between energy and entropy into an optimization schedule (Kirkpatrick et al. 1983); and by 1985 the Boltzmann machine placed Hopfield’s network at a finite temperature and learned its couplings from data (Ackley et al. 1985). The same trade-off resurfaced as the evidence lower bound (Dayan et al. 1995; Neal and Hinton 1998), as the contrastive objectives of energy-based learning (LeCun et al. 2006), and as the temperature parameter of every language-model decoder. In 2024 the Nobel Prize in Physics was awarded to Hopfield and Hinton for this line of work (The Royal Swedish Academy of Sciences 2024). Chapter 3 develops the single equation, \(F = U - TS\), that runs beneath all of it.
We study three directions that are connected:
- Maximizing entropy to obtain the least-biased model consistent with given constraints. This yields exponential-family / Gibbs distributions and, as we will see, a principled framework for valuation problems in machine learning (Bian et al. 2022).
- Minimizing entropy to sharpen a model’s own predictions. When applied to large language models in a latent semantic space, this idea enables fully unsupervised elicitation of reasoning capabilities.
- Minimaxing entropy to decide which features a model should include at all. The optimal features are those whose maximum-entropy model has the minimum entropy, which makes model selection a well-posed problem (Carcamo et al. 2025).
1.2 Roadmap
- Part I — Foundations surveys the thermodynamic, information-theoretic, quantum, and psychological notions of entropy, develops the maximum-entropy principle together with the free-energy relationship \(F = U - TS\) that connects entropy to energy, and introduces energy-based models and the minimum-description-length view that leads to minimax entropy.
- Part II — Methods develops maximum-entropy, minimum-entropy, and minimax-entropy learning, together with energy-based reasoning / guidance / refinement.
- Part III — Applications connects these ideas to scientific foundation models and AI-for-science.