Singular Learning Theory: Part II—Volume scaling and the Real Log Canonical Threshold
Sublevel volume, the RLCT, and phase transitions between competing basins.
published
brookmachine learningsingular learning theorysltinterpretability
In Part I, we saw how symmetries and redundant units in neural networks turn isolated minima into flat troughs and intersecting zero-loss surfaces , where the Fisher information matrix becomes singular and BIC’s volume penalty breaks down. To analyze such singular models without relying on quadratic approximations, Sumio Watanabe developed Singular Learning Theory (SLT), replacing parameter counting with a direct measurement of volume near the optimal set.
Volume scaling and the Real Log Canonical Threshold
Imagine filling the KL landscape with water up to a tiny error level , and measuring the prior volume of the submerged region—the fraction of all parameter configurations whose error falls below :
Now watch what happens as we drain the water toward zero (). For a sufficiently smooth (real-analytic) KL divergence and a smooth, positive prior on a bounded (compact) parameter space, Watanabe proved (using Hironaka’s theorem on resolution of singularities) that this submerged volume shrinks according to a power law:
where the leading exponent is the Real Log Canonical Threshold (RLCT) and the integer is its multiplicity.
For readers skipping the asymptotics, the intuition is direct: measures how rapidly the pool of good weights dries up as we demand higher accuracy. A large means the volume of near-optimal parameters vanishes quickly because most knobs are tightly pinned down; a small means a broad, forgiving neighborhood of parameters keeps the error low.
Two simple examples illustrate how both generalizes parameter counting and reveals why singularities are geometrically cheap.
Example (regular -parameter model). Suppose near , with smooth and positive at the origin. The condition defines a -dimensional ball of radius , whose volume scales as . Reading off the exponent gives and . Whenever a model is regular, the RLCT reduces identically to half the parameter count.
Example (two-parameter cross-singularity). Now consider with a uniform prior on , capturing the intersecting zero-loss lines and of a redundant neuron. Compare its shape directly with a regular two-parameter bowl (): the regular bowl confines to a shrinking circular disk of radius , whose area scales as (). By contrast, forms a cross-shaped region whose long hyperbolic arms stretch all the way out along both axes, capturing much more area near the zeros.
To verify the scaling explicitly, fourfold symmetry reduces the prior integral of over to integrating over the region in the first quadrant . For , we split at : when , any satisfies ; when , we require . Thus
As , the logarithmic term dominates, so . Even though the model still has parameters, its RLCT drops in half from to , with multiplicity .
More generally, different directions in parameter space contribute differently to :
- Active quadratic directions () contribute each, just as in regular models.
- Higher-order degenerate directions ( for ) flatten the valley walls and contribute (for instance, for quartic walls).
- Intersecting slack subspaces (such as ) share fractional contributions ( across the pair rather than ).
- Exact continuous symmetry directions, along which never changes, contribute .
In algebraic geometry, resolution of singularities constructs coordinate transformations that untangle intersecting creases into simple power-law products (monomials) from which can be read off directly. For empirical machine learning, the key conceptual takeaway is simpler: automatically discounts symmetries and idle capacity, so acts as the model’s effective parameter count—roughly how many rigid classical knobs this singular geometry behaves like.
Free energy and competing basins within a network
Plugging this volume-scaling law into the partition function yields Watanabe’s asymptotic expansion for the Bayesian free energy as :
Compare this directly with BIC from Part I: the raw dimension penalty is replaced by the geometric penalty , and the singular behavior that broke is now absorbed cleanly into and . Crucially, because depends only on how the sublevel volume shrinks as , it is invariant to adding redundant parameters or reparametrizing symmetries: doubling the width of a network with duplicate neurons inflates , while remains bounded by the effective complexity of the underlying input-output map.
Notice that the same free-energy decomposition applies not only across separate model families, but also across competing neighborhoods inside a single network. For any local basin , we can restrict the partition function to and define its local free energy
which satisfies (up to lower-order terms) in terms of the local optimum and local RLCT inside .
This gives researchers a precise language for comparing qualitatively different solutions to the same training task. Consider training a transformer on a structured problem such as modular arithmetic, where two distinct basins both fit the training data nearly perfectly:
- A memorization basin (, around ): the network fits individual training examples through brittle, ad-hoc combinations of weights. Because most parameters are pinned down to hold specific associations, little slack remains and is large.
- A circuit basin (, around ): the network implements a compact, reusable algorithm in a small subnetwork, leaving the remaining units free to sit on flat, intersecting zero-planes. Because unused capacity adds little to volume contraction, is substantially smaller.
The difference between their local free energies is
Because the posterior probability of basin is with global partition function , the ratio of posterior mass between the two basins is
Once both basins fit the training set equally well (), the training-loss gap vanishes and this ratio scales as . Even a modest complexity gap of on training examples gives the circuit basin times more posterior mass than the memorizing solution. Simpler internal representations dominate not merely because we prefer them aesthetically, but because they occupy exponentially larger volumes of parameter space.
Two-stage training dynamics and phase transitions
Although SGD does not literally sample from the Bayesian posterior, this balance between training fit () and parameter-space volume () provides a useful physical analogy for two-stage dynamics and abrupt transitions observed during neural network training:
- Error-driven descent ( dominates): Early in training, is large and gradients pull the network toward whichever basin lowers training error fastest. Superficial heuristics or memorization pathways often provide the cleanest initial gradient signal, pulling weights into a high- basin.
- Entropic drift ( dominates): Once training loss approaches its floor, gradients along memorized directions weaken. Persistent mini-batch noise causes SGD to explore rather than stay pinned in narrow canyons, nudging trajectories out of steep, high- valleys into adjacent, lower- basins of far greater volume.
Because dropping requires reorganizing weights so that redundant units release onto intersecting zero-planes, escaping a high- basin often shows up externally as a sharp phase transition—such as delayed generalization in grokking, or the sudden emergence of induction heads during language model pretraining.
In Part III, we turn this theoretical picture into a direct empirical probe—using SGLD to measure the local geometry around the checkpoints that optimization actually visits.