2 Entropy and the Maximum Entropy Principle
2.1 The many faces of entropy
Entropy was invented three separate times — for heat engines in the 1860s, for quantum ensembles in the 1920s, and for telephone lines in the 1940s — and has been rediscovered many times since. The names, the units, and the motivating problems all differ, but each version measures the same thing: given what we have specified about a system, how much remains unspecified. This section walks through those notions before the rest of the book settles on one of them.
2.1.1 Information: Shannon entropy
For a discrete random variable \(X\) with probability mass function \(p(x)\), the Shannon entropy is
\[ H(p) = -\sum_{x} p(x) \log p(x), \]
which quantifies the average uncertainty (in nats or bits) of \(X\) (Shannon 1948). Entropy is maximized by the uniform distribution, where it equals \(\log |\mathcal{X}|\), and minimized (equal to zero) by a deterministic one.
This functional form is not an arbitrary choice. Shannon showed that three requirements — continuity in the probabilities, monotonicity in the number of equally likely outcomes, and consistency when outcomes are grouped — determine \(H\) uniquely up to the base of the logarithm. It also carries a concrete operational meaning: \(H(p)\) is the expected code length of an optimal lossless encoding of samples from \(p\) — exactly, in the limit of coding long blocks, and to within one bit if each symbol must be coded on its own. Uncertainty and description length are one quantity seen from two sides, a duality that returns in Chapter 7.
2.1.2 Heat: thermodynamic entropy
The original entropy is older and, on its face, unrelated to information. Studying the irreversibility of heat flow, Clausius defined a state function whose differential along a reversible path is
\[ dS = \frac{\delta Q_{\text{rev}}}{T}, \]
and named it from the Greek τροπή, transformation, deliberately shaping the word to rhyme with energy because the two quantities seemed to him so closely related (Clausius 1865). That 1865 paper closes with the two sentences that became the standard statement of the laws of thermodynamics: the energy of the universe is constant, and its entropy tends towards a maximum.
Boltzmann supplied the microscopic reading. A macrostate — a temperature, a pressure — is compatible with an enormous number \(W\) of microstates, and entropy simply counts them (Boltzmann 1877):
\[ S = k_B \log W . \]
Gibbs then generalized the count to microstates that are not equally likely (Gibbs 1902):
\[ S = -k_B \sum_i p_i \log p_i . \]
The Gibbs expression and the Shannon expression are the same formula, separated only by the constant \(k_B\) that carries the physical units. The coincidence is not accidental, and Jaynes turned it into the argument developed in Section 2.2: thermodynamic entropy is the Shannon entropy of our ignorance about the microstate, which makes equilibrium statistical mechanics a problem of inference rather than of mechanics (Jaynes 1957).
2.1.3 Quantum states: von Neumann entropy
Quantum mechanics replaces a probability distribution with a density matrix \(\rho\), a positive semidefinite operator of unit trace. Von Neumann extended entropy to this setting in 1927, two decades before Shannon (Neumann 1927):
\[ S(\rho) = -\operatorname{Tr}(\rho \log \rho). \]
Written in the eigenbasis of \(\rho\), this reduces to the Shannon entropy of its eigenvalues, so the same functional form reappears with a different argument. It vanishes exactly for pure states, which are completely specified, and is maximal for the maximally mixed state.
The quantum case does break the analogy in one instructive place. Classically, conditional entropy is never negative: learning \(Y\) cannot leave us knowing more than everything about \(X\). Quantum mechanically it can be — the joint state may be more sharply specified than either of its parts. A negative value therefore certifies entanglement, though the implication runs one way only: entangled states with non-negative conditional entropy exist.
2.1.4 One knob, a family: Rényi and Tsallis
Shannon’s requirements can be relaxed, and each relaxation produces a one-parameter family. Rényi kept additivity over independent systems but dropped the requirement that entropy be a linear average of \(-\log p\) (Rényi 1961):
\[ H_\alpha(p) = \frac{1}{1-\alpha} \log \sum_x p(x)^\alpha , \qquad \alpha > 0, \; \alpha \neq 1 . \]
The order \(\alpha\) is a knob controlling how much weight rare outcomes receive. The limit \(\alpha \to 1\) recovers Shannon entropy; \(\alpha = 0\) gives the Hartley entropy \(\log|\operatorname{supp} p|\), which counts possibilities and ignores their probabilities; \(\alpha = 2\) gives the collision entropy
\[ H_2(p) = -\log \sum_x p(x)^2 , \]
whose argument \(\sum_x p(x)^2\) is the probability that two independent draws coincide — a quantity that returns as a training objective in Section 6.5; and \(\alpha \to \infty\) gives the min-entropy
\[ H_\infty(p) = -\log \max_x p(x), \]
the worst-case measure used in cryptography, where what matters is not average uncertainty but the chance that an adversary guesses right on the first try.
A warning about that last name, since both readings appear later in this book. The min-entropy \(H_\infty\) is a specific functional, the most pessimistic member of the Rényi family. It should not be confused with the minimum-entropy principle of Chapter 6, which is not a functional at all but a class of learning objectives that reduce the entropy — usually the Shannon or the collision entropy — of a model’s predictive distribution. The two are unrelated, and the shared prefix is an accident of terminology in two literatures that developed apart.
Tsallis relaxed the other requirement, giving up additivity itself (Tsallis 1988):
\[ S_q(p) = \frac{1}{q-1}\Big(1 - \sum_x p(x)^q\Big). \]
Here \(q \to 1\) again recovers the Boltzmann–Gibbs form, but for \(q \neq 1\) two independent systems combine as \(S_q(A,B) = S_q(A) + S_q(B) + (1-q)\,S_q(A)\,S_q(B)\). This non-extensivity was introduced to describe systems with long-range interactions and multifractal structure, where the usual assumption that entropy scales with system size fails.
Both families are useful in their own domains, and both make the same point by contrast: Shannon entropy is the unique member satisfying the full set of Shannon–Khinchin axioms, including the chain rule that makes conditional entropy and mutual information well behaved. That is why it is the form used throughout the remainder of this book.
2.1.5 Minds: psychological entropy
The most surprising reappearance is in psychology. Hirsh, Mar, and Peterson propose the entropy model of uncertainty, which applies Shannon’s formula directly to the human information system (Hirsh et al. 2012). At any moment an organism faces a set of competing affordances: possible interpretations of the sensory input, and possible actions to take. The model treats these as a subjectively weighted probability distribution and defines psychological entropy as its Shannon entropy.
The reading is unusually literal. A familiar situation places nearly all the weight on one dominant affordance, giving low entropy and a fast habitual response. An unfamiliar or ambiguous one spreads the weight across many competing frames, giving high entropy and sustained neural competition — which, the authors argue, is what anxiety is subjectively, with anterior cingulate activity and noradrenaline release as its physiological correlates.
Two features of the account matter for what follows. First, goals act as constraints: adopting a goal reweights the distribution towards the actions that serve it, and thereby lowers entropy. This is structurally the same move as imposing moment constraints in Section 2.2, since a constraint is whatever narrows the space of live possibilities. Second, the model makes entropy reduction a drive rather than a calculation. That idea runs from Schrödinger’s argument that organisms persist by exporting entropy to their environment (Schrödinger 1944) through to the free-energy principle (Friston 2010), and it is the biological precedent for the minimum-entropy learning objectives of Chapter 6.
2.1.6 One form, many readings
| Notion | Probabilities over | Definition | Origin |
|---|---|---|---|
| Thermodynamic | no explicit distribution | \(dS = \delta Q_{\text{rev}} / T\) | Clausius, 1865 |
| Statistical | microstates, equally likely | \(k_B \log W\) | Boltzmann, 1877 |
| Gibbs | microstates, unequally likely | \(-k_B \sum_i p_i \log p_i\) | Gibbs, 1902 |
| Information | a random variable or message source | \(-\sum_x p(x) \log p(x)\) | Shannon, 1948 |
| Quantum | eigenvalues of a density matrix | \(-\operatorname{Tr}(\rho \log \rho)\) | von Neumann, 1927 |
| Rényi | any distribution, at order \(\alpha\) | \(\frac{1}{1-\alpha} \log \sum_x p(x)^\alpha\) | Rényi, 1961 |
| Tsallis | non-extensive systems, at order \(q\) | \(\frac{1}{q-1}(1 - \sum_x p(x)^q)\) | Tsallis, 1988 |
| Psychological | competing perceptions and actions | \(-\sum_x p(x) \log p(x)\) | Hirsh et al., 2012 |
Two observations follow. The first is mathematical: almost every row is the same functional \(-\sum p \log p\), differing only in what the probabilities range over and what constant multiplies the result. Entropy is less a family of analogies than a single quantity applied to different objects.
The second is interpretive, and it is the one that does work in this book. The same number reads as dispersal of energy to a physicist, as expected code length to an engineer, and as felt uncertainty to a psychologist. Those readings are what make the two principles that follow feel necessary rather than arbitrary: maximizing entropy is being honest about what we have not specified, and minimizing it is committing to what we have.
2.2 The maximum entropy principle
Given constraints — typically moment constraints of the form \(\mathbb{E}_p[f_k(X)] = \mu_k\) — the maximum entropy distribution solves
\[ \max_{p}\; H(p) \quad \text{s.t.} \quad \sum_x p(x) = 1,\;\; \mathbb{E}_p[f_k(X)] = \mu_k . \]
Its solution is the Gibbs / Boltzmann distribution
\[ p(x) = \frac{1}{Z(\boldsymbol{\lambda})}\, \exp\!\Big(\textstyle\sum_k \lambda_k f_k(x)\Big), \]
where \(\boldsymbol{\lambda}\) are Lagrange multipliers and \(Z\) is the partition function. This is the least-biased distribution consistent with what we know (Jaynes 1957).
The single most important instance of this framework is the one in which the constrained feature is the energy of a physical system. That case, worked out in Chapter 3, is what turns the multipliers above into a temperature and connects the whole construction to thermodynamics.
2.3 Minimum description length and minimax entropy
The maximum-entropy principle can be derived rather than postulated. Under the minimum description length (MDL) view (Grünwald 2007), a model \(P\) encodes each state with a code word of length \(-\log P(x)\), and its description length is the expected code length across all datasets consistent with the measured features. Among models matching those features, description length is minimized by the maximum-entropy model (Feder 1986).
MDL also poses a question that maximum entropy alone cannot answer: which features should be measured in the first place? For maximum-entropy models the description length coincides with the model’s own entropy,
\[ L(P_F) = S(P_F), \]
so the best feature set \(F\) is the one whose maximum-entropy model has the smallest entropy. This is the minimax entropy principle, developed in Chapter 7.
2.4 Why it matters for learning
The maximum-entropy principle explains why exponential families are ubiquitous, grounds the Boltzmann machine in statistical physics, and — as developed in Bian et al. (2022) — provides a principled basis for defining valuations in machine learning. The next chapter supplies the missing half of the picture: what happens when entropy is traded off against energy, and what the exchange rate between them turns out to be.