Entropy-Based Learning

Principles, Methods, and Scientific Applications

An introduction to modern entropy- and energy-based learning, spanning the maximum-entropy, minimum-entropy, and minimax-entropy principles, and their applications to scientific and reasoning-oriented AI.

Author

Yatao Bian

Released

August 2, 2026

Last updated

October 6, 2026

Preface

Welcome to Entropy-Based Learning. This short book is an introductory, evolving companion to a research program on modern entropy- and energy-based learning, developed at the Blue Whale Lab led by Yatao Bian.

Entropy is one of the deepest bridges between physics, information theory, and machine learning — a single quantity reinvented, again and again, for heat engines, telephone lines, quantum ensembles, and even theories of human anxiety. Three complementary principles organize the material in this book:

  • The Maximum Entropy Principle — among all models consistent with the observed constraints, prefer the one that is maximally noncommittal (highest entropy). This principle underlies statistical physics and Boltzmann machines, and it defines a family of learning methods that includes a principled class of game-theoretic valuations, neural set functions learned from their optimal subsets, and maximum-entropy and inverse reinforcement learning. With a pretrained model as the base measure, the same family contains energy-based guidance at inference time and post-training methods such as RLHF, GRPO, and DPO.
  • The Minimum Entropy Principle — when a model’s own predictions can serve as a training signal, reducing predictive uncertainty can elicit capabilities such as reasoning, without external supervision. It elicits but does not add capability, since the training loop never touches ground truth, and, left unchecked, it collapses the model onto one confident answer, right or wrong.
  • The Minimax Entropy Principle — maximize entropy given the features a model includes, then choose the features that make that entropy as small as possible (Zhu et al. 1997; Carcamo et al. 2025). This couples the two principles above, and the same nesting answers two questions: what to measure, by choosing the features that give the shortest description of the data, and what to commit to, by minimizing the entropy of a prediction, which is what a softmax classifier with a linear head already does.

Binding them together is the free energy \(F = U - TS\), in which temperature \(T\) sets the exchange rate between energy and entropy. It is the equation that carried statistical physics into machine learning, and it makes the three principles above three settings of one dial rather than three separate doctrines.

How complete is this draft?

WarningWork in Progress

This book is an ongoing project, released early so you can follow its development. Most chapters are incomplete. The table below shows the current status of each chapter. Please keep this in mind before quoting or citing any part of the text, or before judging drafts that are still being written.

Chapter Status
1 — Introduction early draft
2 — Entropy and the Maximum Entropy Principle mostly complete
3 — Energy, Entropy, and Free Energy mostly complete
4 — Energy-Based Models mostly complete
5 — Maximum-Entropy Learning mostly complete
6 — Minimum-Entropy Learning mostly complete
7 — Minimax-Entropy Learning first full draft
8 — Energy-Based Guidance and Refinement first full draft
9 — Maximum-Entropy Post-Training first full draft
10 — Entropy-Based Learning for Science early draft
11 — Summary early draft

“Mostly complete” means the argument is in place and has been checked, though details may still move. “First full draft” means the chapter is written end to end but has not yet been checked with the same care. “Early draft” means little more than a scaffold. The table will be updated as chapters mature. Feedback and contributions are welcome at any stage.

A printable PDF of the current draft is available from the download button in the sidebar, or directly as Entropy-Based-Learning.pdf.

To learn more about the underlying research, visit https://yataobian.com/ and the Blue Whale Lab.

Citation

If you found this book useful, please cite it as:

@book{bian2026entropy,
  title     = {Entropy-Based Learning},
  author    = {Yatao Bian},
  year      = {2026},
  publisher = {Blue Whale Lab},
  url       = {https://yataobian.github.io/entropy-based-learning/}
}

Or, for attribution in text: Bian, Yatao. 2026. Entropy-Based Learning. Blue Whale Lab. https://yataobian.github.io/entropy-based-learning/.