Contents

• 1. Introduction
  • 1.1 Why entropy?
  • 1.2 Roadmap

Part I — Foundations

• 2. Entropy and the Maximum Entropy Principle
  • 2.1 The many faces of entropy
    • 2.1.1 Information: Shannon entropy
    • 2.1.2 Heat: thermodynamic entropy
    • 2.1.3 Quantum states: von Neumann entropy
    • 2.1.4 One knob, a family: Rényi and Tsallis
    • 2.1.5 Minds: psychological entropy
    • 2.1.6 One form, many readings
  • 2.2 Why these formulas and no others: the axioms
    • 2.2.1 The load-bearing axiom
    • 2.2.2 Loosening one axiom at a time
    • 2.2.3 The other faces, axiomatized
    • 2.2.4 From entropy to diversity
    • 2.2.5 Similarity
    • 2.2.6 What axioms cannot settle
  • 2.3 The maximum entropy principle
  • 2.4 Minimum description length and minimax entropy
  • 2.5 Why it matters for learning
• 3. Energy, Entropy, and Free Energy
  • 3.1 Maximum entropy with an energy constraint
  • 3.2 Free energy
  • 3.3 What entropy buys you
    • 3.3.1 Isothermal expansion
    • 3.3.2 The rubber band
    • 3.3.3 Melting
  • 3.4 From spins to networks
  • 3.5 How this reached AI
  • 3.6 The temperature dial
  • 3.7 Why it matters for learning
• 4. Energy-Based Models
  • 4.1 The energy view of inference
  • 4.2 Why learn an energy at all?
  • 4.3 Learning is shaping a landscape
  • 4.4 Strategy one: sample the pull-up
  • 4.5 Strategy two: match slopes, not heights
  • 4.6 Strategy three: learn by telling things apart
  • 4.7 One family, three estimators
  • 4.8 Latent variables, and the free energy again
  • 4.9 The modern landscape
  • 4.10 Where this sits in the book
  • 4.11 Takeaway

Part II — Methods

• 5. Maximum-Entropy Learning
  • 5.1 What this chapter means by maximum-entropy learning
    • 5.1.1 Representation: maximum-entropy models in two forms
    • 5.1.2 Inference: free-energy minimization over a tractable family
    • 5.1.3 Learning: fitting the energy through the inference
    • 5.1.4 The family, and a running example
  • 5.2 Graphical models as an explanatory family
  • 5.3 A worked example: energy-based cooperative games
    • 5.3.1 Why this is useful
  • 5.4 Discrete maximum-entropy learning: set functions under the optimal subset oracle
    • 5.4.1 The optimal subset oracle
    • 5.4.2 Learning through mean-field inference: EquiVSet
    • 5.4.3 Which statistics may the energy depend on? INSET
    • 5.4.4 What the strand adds to the maximum-entropy programme
  • 5.5 Maximum-entropy reinforcement learning: energies over trajectories
    • 5.5.1 A Gibbs distribution over trajectories
    • 5.5.2 Soft Bellman equations as free energies
    • 5.5.3 Inverse reinforcement learning: the optimal subset oracle on trajectories
  • 5.6 What the family adds, and where it stops
    • 5.6.1 Maximum likelihood as a saddle-point problem
    • 5.6.2 Maximum entropy inside, minimum entropy outside: the bridge to minimax entropy
    • 5.6.3 Where the family stops
  • 5.7 Takeaway
• 6. Minimum-Entropy Learning
  • 6.1 Two regimes, and a warning about the name
  • 6.2 From word-level to meaning-level uncertainty
  • 6.3 Entropy minimisation as an objective: a short history
  • 6.4 EMPO: reward agreement, not correctness
  • 6.5 What “reward agreement” actually optimises
  • 6.6 The free-energy view, and why entropy collapses
  • 6.7 The information ceiling
  • 6.8 Schrödinger, read carefully
  • 6.9 Prigogine, and an analogy that does not hold
  • 6.10 So why call it minimum-entropy learning?
  • 6.11 When agreement means truth, and when it does not
  • 6.12 Takeaway
• 7. Minimax-Entropy Learning
  • 7.1 The minimax entropy principle: what to measure
    • 7.1.1 Three equivalent views
    • 7.1.2 Solving it
    • 7.1.3 Implicit minimax entropy in energy-based models
  • 7.2 Minimax entropy on the joint space: modelling and predicting
    • 7.2.1 Two questions, one principle
    • 7.2.2 Classification is conditional minimax entropy
    • 7.2.3 A classifier is a joint maximum-entropy model
  • 7.3 When the label is missing
    • 7.3.1 A temperature on the label
    • 7.3.2 What the density term buys, and what it does not
  • 7.4 Boundaries, evidence, and a name clash
  • 7.5 Takeaway
• 8. Energy-Based Guidance and Refinement
  • 8.1 The tilted reference
  • 8.2 Continuous guidance
  • 8.3 Discrete guidance: residual energies over a proposal
  • 8.4 Verifiers as energies, best-of-N, and sequential Monte Carlo
  • 8.5 Refinement and reasoning as energy minimization
  • 8.6 Guidance on the three levels
  • 8.7 Takeaway
• 9. Maximum-Entropy Post-Training
  • 9.1 From the reference to its tilt
  • 9.2 Amortizing the tilt
  • 9.3 Learning the energy
  • 9.4 Self-referential energies
  • 9.5 The continuous case
  • 9.6 Post-training on the three levels, and where it stops
  • 9.7 Takeaway

Part III — Applications

• 10. Entropy-Based Learning for Science
  • 10.1 Two synergistic directions
  • 10.2 Molecular foundation models
  • 10.3 Outlook
• 11. Summary
• References