Singular Learning Theory: Part II—Volume scaling and the Real Log Canonical Threshold

Sublevel volume, the RLCT, and phase transitions between competing basins.

published

brookmachine learningsingular learning theorysltinterpretability

In Part I, we saw how symmetries and redundant units in neural networks turn isolated minima into flat troughs and intersecting zero-loss surfaces W0={w∈W:K(w)=0}, where the Fisher information matrix I(w) becomes singular and BIC’s d2log⁡n volume penalty breaks down. To analyze such singular models without relying on quadratic approximations, Sumio Watanabe developed Singular Learning Theory (SLT), replacing parameter counting with a direct measurement of volume near the optimal set.

Volume scaling and the Real Log Canonical Threshold

Imagine filling the KL landscape K(w) with water up to a tiny error level ϵ>0, and measuring the prior volume of the submerged region—the fraction of all parameter configurations whose error falls below ϵ:

V(ϵ)=∫K(w)<ϵφ(w)dw.

Now watch what happens as we drain the water toward zero (ϵ→0+). For a sufficiently smooth (real-analytic) KL divergence K(w) and a smooth, positive prior φ on a bounded (compact) parameter space, Watanabe proved (using Hironaka’s theorem on resolution of singularities) that this submerged volume shrinks according to a power law:

V(ϵ)∝ϵλ(−log⁡ϵ)m−1,

where the leading exponent λ>0 is the Real Log Canonical Threshold (RLCT) and the integer m≥1 is its multiplicity.

For readers skipping the asymptotics, the intuition is direct: λ measures how rapidly the pool of good weights dries up as we demand higher accuracy. A large λ means the volume of near-optimal parameters vanishes quickly because most knobs are tightly pinned down; a small λ means a broad, forgiving neighborhood of parameters keeps the error low.

Two simple examples illustrate how λ both generalizes parameter counting and reveals why singularities are geometrically cheap.

Example (regular d-parameter model). Suppose K(w)=w12+w22+⋯+wd2 near w0=0, with φ smooth and positive at the origin. The condition K(w)<ϵ defines a d-dimensional ball of radius ϵ1/2, whose volume scales as (ϵ1/2)d=ϵd/2. Reading off the exponent gives λ=d/2 and m=1. Whenever a model is regular, the RLCT reduces identically to half the parameter count.

Example (two-parameter cross-singularity). Now consider K(w1,w2)=w12w22 with a uniform prior φ(w1,w2)=1/4 on W=[−1,1]2, capturing the intersecting zero-loss lines w1=0 and w2=0 of a redundant neuron. Compare its shape directly with a regular two-parameter bowl (w12+w22<ϵ): the regular bowl confines (w1,w2) to a shrinking circular disk of radius ϵ, whose area scales as (ϵ)2=ϵ1 (λ=1). By contrast, |w1w2|<ϵ forms a cross-shaped region whose long hyperbolic arms stretch all the way out along both axes, capturing much more area near the zeros.

To verify the scaling explicitly, fourfold symmetry reduces the prior integral of 1/4 over [−1,1]2 to integrating 1 over the region w1w2<ϵ in the first quadrant [0,1]2. For 0<ϵ<1, we split at w1=ϵ: when 0≤w1<ϵ, any w2∈[0,1] satisfies w1w2<ϵ; when ϵ≤w1≤1, we require w2<ϵ/w1. Thus

V(ϵ)=∫0ϵ1dw1+∫ϵ1ϵw1dw1=ϵ1/2+12ϵ1/2(−log⁡ϵ).

As ϵ→0+, the logarithmic term dominates, so V(ϵ)∝ϵ1/2(−log⁡ϵ). Even though the model still has d=2 parameters, its RLCT drops in half from λ=1 to λ=1/2, with multiplicity m=2.

More generally, different directions in parameter space contribute differently to λ:

  • Active quadratic directions (K∝wi2) contribute 1/2 each, just as in regular models.
  • Higher-order degenerate directions (K∝wi2k for k≥2) flatten the valley walls and contribute 1/(2k) (for instance, 1/4 for quartic walls).
  • Intersecting slack subspaces (such as K∝w12w22) share fractional contributions (1/2 across the pair rather than 1/2+1/2=1).
  • Exact continuous symmetry directions, along which K(w) never changes, contribute 0.

In algebraic geometry, resolution of singularities constructs coordinate transformations that untangle intersecting creases into simple power-law products (monomials) from which λ can be read off directly. For empirical machine learning, the key conceptual takeaway is simpler: λ automatically discounts symmetries and idle capacity, so 2λ acts as the model’s effective parameter count—roughly how many rigid classical knobs this singular geometry behaves like.

Free energy and competing basins within a network

Plugging this volume-scaling law into the partition function Zn yields Watanabe’s asymptotic expansion for the Bayesian free energy as n→∞:

Fn=nLn(w0)+λlog⁡n−(m−1)log⁡log⁡n+𝒪p(1).

Compare this directly with BIC from Part I: the raw dimension penalty d2log⁡n is replaced by the geometric penalty λlog⁡n, and the singular behavior that broke log⁡det⁡I(w0) is now absorbed cleanly into λ and m. Crucially, because λ depends only on how the sublevel volume V(ϵ) shrinks as ϵ→0+, it is invariant to adding redundant parameters or reparametrizing symmetries: doubling the width of a network with duplicate neurons inflates d, while λ remains bounded by the effective complexity of the underlying input-output map.

Notice that the same free-energy decomposition applies not only across separate model families, but also across competing neighborhoods inside a single network. For any local basin U⊂W, we can restrict the partition function to U and define its local free energy

Fn(U)=−log⁡∫Uexp⁡(−nLn(w))φ(w)dw,

which satisfies Fn(U)≈nLn(wU)+λUlog⁡n (up to lower-order terms) in terms of the local optimum wU and local RLCT λU inside U.

This gives researchers a precise language for comparing qualitatively different solutions to the same training task. Consider training a transformer on a structured problem such as modular arithmetic, where two distinct basins both fit the training data nearly perfectly:

  1. A memorization basin (Umem, around wmem): the network fits individual training examples through brittle, ad-hoc combinations of weights. Because most parameters are pinned down to hold specific associations, little slack remains and λmem is large.
  2. A circuit basin (Ucirc, around wcirc): the network implements a compact, reusable algorithm in a small subnetwork, leaving the remaining units free to sit on flat, intersecting zero-planes. Because unused capacity adds little to volume contraction, λcirc is substantially smaller.

The difference between their local free energies is

ΔFn=Fn(Umem)−Fn(Ucirc)≈n(Ln(wmem)−Ln(wcirc))+(λmem−λcirc)log⁡n.

Because the posterior probability of basin U is P(U|Dn)=1Zn∫Uexp⁡(−nLn(w))φ(w)dw=1Zne−Fn(U) with global partition function Zn, the ratio of posterior mass between the two basins is

P(Ucirc|Dn)P(Umem|Dn)=e−Fn(Ucirc)e−Fn(Umem)=eΔFn.

Once both basins fit the training set equally well (Ln(wmem)≈Ln(wcirc)), the training-loss gap vanishes and this ratio scales as nλmem−λcirc. Even a modest complexity gap of λmem−λcirc=10 on n=10,000 training examples gives the circuit basin 10,00010=1040 times more posterior mass than the memorizing solution. Simpler internal representations dominate not merely because we prefer them aesthetically, but because they occupy exponentially larger volumes of parameter space.

Two-stage training dynamics and phase transitions

Although SGD does not literally sample from the Bayesian posterior, this balance between training fit (nLn) and parameter-space volume (λlog⁡n) provides a useful physical analogy for two-stage dynamics and abrupt transitions observed during neural network training:

  • Error-driven descent (nLn dominates): Early in training, Ln(w) is large and gradients pull the network toward whichever basin lowers training error fastest. Superficial heuristics or memorization pathways often provide the cleanest initial gradient signal, pulling weights into a high-λ basin.
  • Entropic drift (λlog⁡n dominates): Once training loss approaches its floor, gradients along memorized directions weaken. Persistent mini-batch noise causes SGD to explore rather than stay pinned in narrow canyons, nudging trajectories out of steep, high-λ valleys into adjacent, lower-λ basins of far greater volume.

Because dropping λ requires reorganizing weights so that redundant units release onto intersecting zero-planes, escaping a high-λ basin often shows up externally as a sharp phase transition—such as delayed generalization in grokking, or the sudden emergence of induction heads during language model pretraining.

In Part III, we turn this theoretical picture into a direct empirical probe—using SGLD to measure the local geometry around the checkpoints that optimization actually visits.