9 Entropy-Based Learning for Science
The ultimate aim of this research program is scientific intelligence — AI that is not only accurate and robust, but increasingly endowed with reasoning capabilities that enable new modes of scientific inquiry. Entropy- and energy-based learning offer principled tools for this goal.
9.1 Two synergistic directions
- Advancing AI with Science. Fundamental scientific and mathematical theories — statistical physics, information theory, the maximum-entropy principle — inspire principled AI methodologies. Energy-based games (Chapter 5), entropy-minimized reasoning (Chapter 6), and minimax-entropy model selection (Chapter 7) are three examples.
- Advancing Science with AI. The same methods accelerate discovery in the sciences, from molecular representation learning to chemical reasoning.
9.2 Molecular foundation models
A concrete thread runs from self-supervised graph transformers for molecular representation learning toward molecule–language models with reasoning capabilities. The trajectory mirrors the broader shift from pretraining–finetuning to LLM-style reasoning, and entropy-based objectives provide label-efficient signals where annotated data are scarce.
9.3 Outlook
A deeper unification may be within reach. The minimax view (Chapter 7) suggests that modelling the world and committing to a decision are two halves of one objective — an idea that echoes the free-energy principle proposed as a unified account of perception and action in the brain (Friston 2010). The quantity that principle asks the brain to minimize is the variational free energy of Chapter 3, which is what makes the correspondence more than a metaphor. Making it precise, and scaling it to LLMs, agents, and multimodal scientific models, is an open and appealing direction.
As the research matures, this book will grow to include worked examples, executable code, and reproducible experiments.
See the Blue Whale Lab and https://yataobian.com/ for the latest developments.