Genefold First Principles: Depletion of Uncertainty

1. Shannon's starting point: information as uncertainty reduction

Claude Shannon's formulation makes entropy the central quantity for a probabilistic source. For a discrete variable $X$ with outcomes $x_i$ and probabilities $p_i$, entropy is:

$$H(X) = - \sum_i p_i \log_2 p_i$$

This is the expected value of surprise, measured in bits. A rare event carries more surprise than a common one, and a perfectly predictable source has zero entropy. If every outcome is equally likely, entropy rises with the number of possible states because there is more uncertainty to resolve.

A useful way to read this is:

• self-information of one event: $I(x_i) = -\log_2 p_i$,
• entropy of the whole source: $H(X) = \mathbb{E}[I(X)]$.

Entropy is not "noise" in the abstract. It is the average amount of uncertainty left before the message arrives.

import math

def shannon_entropy(probs):
    return -sum(p * math.log2(p) for p in probs if p > 0)

print(shannon_entropy([0.5, 0.5]))   # 1.0
print(shannon_entropy([0.8, 0.2]))   # 0.72
print(shannon_entropy([0.99, 0.01])) # 0.08

2. Entropy grows with bits because uncertainty multiplies

If a message contains $N$ independent binary choices and each bit is fair, there are $2^N$ possible messages. Each full message has probability $2^{-N}$, so its information content is:

$$I = -\log_2(2^{-N}) = N$$

The uncertainty of a fair $N$-bit message is exactly $N$ bits. Every additional fair bit doubles the number of possible states, so the receiver must resolve one more bit of uncertainty.

import math
import numpy as np
import matplotlib.pyplot as plt

def binary_entropy(p):
    if p == 0.0 or p == 1.0:
        return 0.0
    return -(p * math.log2(p) + (1 - p) * math.log2(1 - p))

bits = np.arange(1, 21)
p = 0.5
entropy_per_bit = binary_entropy(p)
total_entropy = bits * entropy_per_bit

for n, h in zip(bits[:5], total_entropy[:5]):
    print(f"{n} bit(s): H = {h:.1f} bits")

plt.figure(figsize=(8, 4.5))
plt.plot(bits, total_entropy, marker="o", linewidth=2)
plt.title("Entropy growth for a fair binary source")
plt.xlabel("Number of bits")
plt.ylabel("Total entropy (bits)")
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

For a fair source, the plot is a straight line because each added bit contributes one more bit of uncertainty. If the source becomes biased, the line bends downward: the source becomes more predictable, and the entropy per bit falls below 1.

This is the first clue for neural networks: a good model is one that makes the next token less surprising.

3. Why cross-entropy appears in LLM training

Cross-entropy is the mathematical structure that connects the model's distribution to the data's true distribution. If the true distribution is $p$ and the model distribution is $q$, cross-entropy is:

$$H(p, q) = - \sum_x p(x)\log_2 q(x)$$

Entropy $H(p)$ is the irreducible uncertainty of the data source — the minimum number of bits needed to encode events drawn from $p$. Cross-entropy $H(p, q)$ is the total coding cost when the model uses distribution $q$ to encode events from $p$. It is the structure that tells the optimizer how close $q$ is to $p$. The difference between the two is the KL divergence:

$$D_{KL}(p \| q) = H(p, q) - H(p) \geq 0$$

For language models, the training objective is next-token prediction. The model assigns a probability distribution over the vocabulary, and the loss penalizes low probability on the correct token. If the model gives the true next token probability $q(y)$, the loss is proportional to:

$$-\log q(y)$$

That is the per-sample contribution to cross-entropy. Better predictions mean the model assigns higher probability to the truth, which lowers the coding cost. Since the data's entropy $H(p)$ is fixed, minimizing cross-entropy is equivalent to minimizing the KL divergence — the gap between what the model believes and what the data actually is.

import math

def cross_entropy_single_token(prob_true_token):
    return -math.log2(max(prob_true_token, 1e-15))

examples = [0.95, 0.50, 0.10, 0.01]
for p in examples:
    print(f"q(true)={p:.2f} -> loss={cross_entropy_single_token(p):.3f} bits")

4. Depletion of uncertainty as a training principle

The phrase "depletion of uncertainty" is a lens for understanding modern loss functions. In Shannon's view, receiving a message removes uncertainty. In LLM training, the model's distribution $q$ should converge toward the true distribution $p$ as it learns. Cross-entropy is the structure that measures the cost of the gap; the KL divergence $D_{KL}(p \| q)$ is the gap itself. Training depletes the KL divergence — the extra cost the model pays for not matching the data — driving it toward zero.

That is why loss minimization feels so universal across deep learning:

• entropy $H(p)$ is the irreducible uncertainty of the source,
• cross-entropy $H(p, q)$ measures the total cost of encoding the data with the model's distribution,
• the KL divergence $D_{KL}(p \| q) = H(p, q) - H(p)$ is the gap that training depletes.

Viewed this way, the model is not trying to "be smart" in a vague sense. It is trying to match the true distribution $p$ closely enough that the KL divergence — the price of being wrong — approaches zero.

flowchart LR
    A[Possible states] --> B[Entropy]
    B --> C[Uncertainty]
    C --> D[Observation / message]
    D --> E[Uncertainty depleted]

    F[Model logits] --> G[Softmax distribution]
    G --> H[Cross-entropy loss]
    H --> I[Gradient update]
    I --> J[Less surprise next time]

5. A note on diffusion and ELBO

There is a parallel story in diffusion models. The direct likelihood objective is often intractable, so training uses a lower bound, the ELBO, as the computable route toward the same end. In that setting too, the loss is still about reducing uncertainty step by step: denoising one noisy state at a time until the model's distribution is close to the data distribution.

Whether the system is a next-token predictor or a denoiser, the geometry of learning is the same. The model is learning to explain away surprise.

6. Current literature to follow

A useful recent reference on information-theoretic dilution in high-dimensional systems is arXiv:2504.08807, which frames an "Information Dilution Theorem" linking growing dimensionality to declining metric-information efficiency. It is not about LLM loss directly, but it is relevant because it formalizes how information can become less recoverable as systems grow larger and noisier.

For broader context, the NeurIPS proceedings page is a good entry point for current work across learning theory, representation learning, and generative modeling.

7. Closing idea

The cleanest way to think about entropy, cross-entropy, and LLM loss is this: entropy $H(p)$ is the irreducible uncertainty of the source, cross-entropy $H(p, q)$ is the mathematical structure that measures how close the model's distribution $q$ is to the true distribution $p$, and training depletes the KL divergence $D_{KL}(p \| q) = H(p, q) - H(p)$ — the gap between what the model believes and what the data is. Genefold's engineering perspective starts from that first principle: better models are not just more accurate, they are less surprised by reality.