Entropy-Based Learning

Principles, Methods, and Scientific Applications

An introduction to modern entropy- and energy-based learning, spanning the maximum-entropy, minimum-entropy, and minimax-entropy principles, and their applications to scientific and reasoning-oriented AI.

Author

Yatao Bian

Released

August 2, 2026

Last updated

August 23, 2026

Preface

Welcome to Entropy-Based Learning. This short book is an introductory, evolving companion to a research program on modern entropy- and energy-based learning, developed at the Blue Whale Lab led by Yatao Bian.

Entropy is one of the deepest bridges between physics, information theory, and machine learning — a single quantity reinvented, again and again, for heat engines, telephone lines, quantum ensembles, and even theories of human anxiety. Three complementary principles organize the material in this book:

  • The Maximum Entropy Principle — among all models consistent with the observed constraints, prefer the one that is maximally noncommittal (highest entropy). This principle underlies statistical physics, Boltzmann machines, and a principled class of game-theoretic valuation methods.
  • The Minimum Entropy Principle — when a model’s own predictions can serve as a training signal, reducing predictive uncertainty can elicit capabilities such as reasoning, without external supervision.
  • The Minimax Entropy Principle — maximize entropy given the features a model includes, then choose the features that make that entropy as small as possible. This couples the two principles above and turns model selection itself into a well-posed problem (Zhu et al. 1997; Carcamo et al. 2025).

Binding them together is the free energy \(F = U - TS\), in which temperature \(T\) sets the exchange rate between energy and entropy. It is the equation that carried statistical physics into machine learning, and it makes the three principles above three settings of one dial rather than three separate doctrines.

How complete is this draft?

WarningWork in Progress

This book is an ongoing project, released early so you can follow its development. Most chapters are incomplete. The table below shows the current status of each chapter. Please keep this in mind before quoting or citing any part of the text, or before judging drafts that are still being written.

Chapter Status
1 — Introduction early draft
2 — Entropy and the Maximum Entropy Principle mostly complete
3 — Energy, Entropy, and Free Energy mostly complete
4 — Energy-Based Models mostly complete
5 — Maximum-Entropy Learning early draft
6 — Minimum-Entropy Learning mostly complete
7 — Minimax-Entropy Learning early draft
8 — Energy-Based Reasoning, Guidance, and Refinement early draft
9 — Entropy-Based Learning for Science early draft
10 — Summary early draft

“Mostly complete” means the argument is in place and has been checked, though details may still move. “Early draft” means little more than a scaffold. The table will be updated as chapters mature. Feedback and contributions are welcome at any stage.

A printable PDF of the current draft is available from the download button in the sidebar, or directly as Entropy-Based-Learning.pdf.

To learn more about the underlying research, visit https://yataobian.com/ and the Blue Whale Lab.

Citation

If you found this book useful, please cite it as:

@book{bian2026entropy,
  title     = {Entropy-Based Learning},
  author    = {Yatao Bian},
  year      = {2026},
  publisher = {Blue Whale Lab},
  url       = {https://yataobian.github.io/entropy-based-learning/}
}

Or, for attribution in text: Bian, Yatao. 2026. Entropy-Based Learning. Blue Whale Lab. https://yataobian.github.io/entropy-based-learning/.