1. The problem with treating data as a flat commodity
Most pre-training pipelines still operate on a simple premise: scrape trillions of tokens, shuffle uniformly, scale compute, and hope for broad out-of-distribution generalization. The implicit assumption is that more tokens = more knowledge, with each token carrying equal marginal value.
This assumption breaks under scrutiny. Random noise and trivial repetitions inflate token counts while contributing near-zero learnable structure. Uniform shuffling discards the curriculum signal that structured data naturally provides. And “hope” is not an engineering strategy.
Shannon entropy captures the average uncertainty of a source. But deep learning operates under finite computational budgets — not the unbounded observers of classical information theory. When compute is bounded, not all bits carry the same value.
import numpy as np
from collections import Counter
def shannon_entropy(tokens):
"""Classical Shannon entropy — treats all tokens equally."""
counts = Counter(tokens)
total = len(tokens)
probs = [c / total for c in counts.values()]
return -sum(p * np.log2(p) for p in probs)
# Example: structured vs. random data
structured = ["the", "cat", "sat", "on", "the", "mat"] * 1000
random_noise = np.random.choice(50000, size=6000).tolist()
print(f"Structured entropy: {shannon_entropy(structured):.2f} bits")
print(f"Random noise entropy: {shannon_entropy(random_noise):.2f} bits")
# Both can have similar entropy, but vastly different learnable content2. Epiplexity: isolating usable structural information
Finzi et al. (2026) introduce Epiplexity ($S_T$) to formalize this distinction. The key insight: time-bounded entropy ($H_T$) conflates three distinct quantities:
- Structural information — reusable, transferable computational circuits the model can internalize
- Residual noise — incompressible stochasticity that no amount of compute can resolve
- Trivial redundancy — patterns so simple they're learned in the first few steps
Epiplexity isolates the first component. Formally, for a dataset $D$ and a compute budget $T$:
$$S_T(D) = \underbrace{H_T(D)}_{\text{time-bounded entropy}} - \underbrace{R_T(D)}_{\text{residual noise}} - \underbrace{C_T(D)}_{\text{trivial compressibility}}$$
Where $H_T$ is the entropy estimable within budget $T$, $R_T$ is the portion that remains unpredictable even with optimal computation, and $C_T$ is the portion compressible by trivial algorithms (e.g., n-gram statistics).
The practical implication: two datasets with identical Shannon entropy can have radically different epiplexity — and therefore radically different returns on compute investment.
def estimate_epiplexity(tokens, compute_budget_steps, vocab_size=50000):
"""
Toy estimator for epiplexity components.
Real implementation uses cross-domain scaling laws (Su et al., 2026).
"""
# H_T: entropy estimable within compute budget
h_t = shannon_entropy(tokens)
# C_T: trivial compressibility (n-gram baseline)
bigram_entropy = estimate_bigram_entropy(tokens)
c_t = h_t - bigram_entropy # what n-grams already capture
# R_T: residual noise (lower bound from scaling law extrapolation)
r_t = estimate_residual_noise(tokens, compute_budget_steps)
s_t = h_t - r_t - c_t
return {
"time_bounded_entropy": h_t,
"trivial_compressibility": c_t,
"residual_noise": r_t,
"epiplexity": max(0, s_t)
}
def estimate_bigram_entropy(tokens):
"""Entropy captured by simple bigram statistics."""
bigrams = list(zip(tokens[:-1], tokens[1:]))
counts = Counter(bigrams)
total = len(bigrams)
probs = [c / total for c in counts.values()]
return -sum(p * np.log2(p) for p in probs)
def estimate_residual_noise(tokens, budget):
"""Estimate irreducible noise via scaling law extrapolation."""
# Placeholder: real version fits power law to loss curves
# and extrapolates to infinite compute
return 0.15 * shannon_entropy(tokens) # heuristic3. EpiSelect and EpiGen: operationalizing epiplexity
Su et al. (2026) bridge theory to practice with two mechanisms that use epiplexity as a steering signal.
EpiSelect: dynamic data selection via cross-domain scaling laws
Instead of static filtering or heuristic quality scores, EpiSelect learns a cross-domain scaling law that predicts how much a data domain contributes to downstream capabilities per unit of compute.
class EpiSelect:
"""
Learns scaling exponents per data domain to dynamically
allocate compute to highest-epiplexity sources.
"""
def __init__(self, domains, initial_weights=None):
self.domains = domains
self.weights = initial_weights or {d: 1.0 for d in domains}
self.scaling_history = {d: [] for d in domains}
def update(self, domain_losses, step):
"""
domain_losses: {domain: validation_loss}
Fits power law: loss = a * compute^b + c
Epiplexity proxy = -b (steeper decay = more learnable structure)
"""
for domain, loss in domain_losses.items():
self.scaling_history[domain].append((step, loss))
if step % 1000 == 0: # re-estimate periodically
self._reestimate_weights()
def _reestimate_weights(self):
for domain, history in self.scaling_history.items():
if len(history) < 10:
continue
steps, losses = zip(*history)
# Fit log(loss - c) = log(a) + b * log(steps)
# Simplified: use last two points for exponent estimate
if len(losses) >= 2:
b = np.log(losses[-1] / losses[-2]) / np.log(steps[-1] / steps[-2])
# Higher epiplexity = more negative b (faster learning)
self.weights[domain] = max(0.1, -b)
# Normalize to probability distribution
total = sum(self.weights.values())
self.weights = {d: w / total for d, w in self.weights.items()}
def sample_batch(self, batch_size):
"""Sample domains proportionally to estimated epiplexity."""
domains = list(self.weights.keys())
probs = list(self.weights.values())
return np.random.choice(domains, size=batch_size, p=probs)EpiGen: synthesizing informative data via policy gradients
When natural data is insufficient, EpiGen generates synthetic data by optimizing a loss-reduction buffer — a differentiable proxy for epiplexity gain.
import torch
import torch.nn as nn
class EpiGen(nn.Module):
"""
Policy gradient over synthetic data generator.
Reward = reduction in validation loss on target domains.
"""
def __init__(self, generator, validator, target_domains):
super().__init__()
self.generator = generator # e.g., small LM or diffusion model
self.validator = validator # frozen model evaluating generated data
self.target_domains = target_domains
self.loss_buffer = {d: [] for d in target_domains}
def forward(self, batch_size):
# Generate synthetic batch
synthetic_data = self.generator.generate(batch_size)
# Evaluate on validator (no gradient through validator)
with torch.no_grad():
val_losses = self.validator.evaluate(synthetic_data)
# Reward: negative loss = higher epiplexity
rewards = {d: -val_losses[d] for d in self.target_domains}
# Policy gradient: maximize expected reward
log_probs = self.generator.log_prob(synthetic_data)
loss = -sum(rewards[d] * log_probs for d in self.target_domains)
return loss, rewards
def update_buffer(self, domain, loss):
"""Track loss reduction as epiplexity signal."""
self.loss_buffer[domain].append(loss)
if len(self.loss_buffer[domain]) > 1000:
self.loss_buffer[domain].pop(0)
def epiplexity_signal(self, domain):
"""Smoothed loss reduction rate."""
buf = self.loss_buffer[domain]
if len(buf) < 10:
return 0.0
return -(buf[-1] - buf[0]) / len(buf) # negative = improvement4. Engineering principles: from theory to production
Bridging mathematical clarity to high-throughput systems requires disciplined engineering choices.
4.1 Deconstruct the true signal
Before throwing raw compute at the problem, disentangle residual noise from actionable structural information.
# Production pipeline: epiplexity-aware data loading
class EpiplexityDataLoader:
def __init__(self, datasets, selector: EpiSelect, budget_per_step):
self.datasets = datasets
self.selector = selector
self.budget = budget_per_step
self.step = 0
def __iter__(self):
while True:
# Select domain based on current epiplexity estimates
domain = self.selector.sample_batch(1)[0]
# Yield batch from selected domain
batch = next(self.datasets[domain])
# Update selector with validation signal (async in practice)
if self.step % 100 == 0:
val_losses = self._validate_sample(domain)
self.selector.update(val_losses, self.step)
self.step += 1
yield batch
def _validate_sample(self, domain):
"""Lightweight validation on held-out slice."""
# In production: separate validation shards per domain
return {domain: np.random.random()} # placeholder4.2 Modular, uncompromising architectures
Decouple data orchestration from low-level execution through strict binary contracts.
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Data Sources │────▶│ EpiSelect │────▶│ Training │
│ (domains) │ │ (orchestrator) │ │ Runtime │
└─────────────────┘ └──────────────────┘ └─────────────────┘
│
▼
┌──────────────────┐
│ Telemetry Bus │
│ (binary proto) │
└──────────────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Logging │ │ Alerting │ │ EpiGen │
│ / Metrics│ │ │ │ Synthesizer
└──────────┘ └──────────┘ └──────────┘The orchestrator emits structured telemetry (domain, step, loss, epiplexity estimate, compute cost) over a binary protocol. Downstream consumers — logging, alerting, EpiGen — subscribe without coupling to the orchestrator's internals.
4.3 Validate on synthetic ground truths
Always verify algorithmic mechanics against controlled baselines with known dynamics before scaling.
def test_epiplexity_estimator():
"""Synthetic ground truth: known mixture of structure/noise/redundancy."""
# Domain A: highly structured (low entropy, high epiplexity)
structured = ["A", "B", "C"] * 10000 # deterministic pattern
# Domain B: pure noise (high entropy, zero epiplexity)
noise = np.random.randint(0, 1000, 30000).tolist()
# Domain C: trivial redundancy (low entropy, low epiplexity)
redundant = ["the"] * 30000
estimator = EpiplexityEstimator()
s_structured = estimator.estimate(structured)
s_noise = estimator.estimate(noise)
s_redundant = estimator.estimate(redundant)
# Assertions codify theoretical expectations
assert s_structured["epiplexity"] > s_noise["epiplexity"] * 10
assert s_structured["epiplexity"] > s_redundant["epiplexity"] * 5
assert s_noise["residual_noise"] > s_structured["residual_noise"] * 5
assert s_redundant["trivial_compressibility"] > s_structured["trivial_compressibility"] * 5
print("✓ Synthetic ground truth validated")
test_epiplexity_estimator()5. Current literature to follow
| Paper | Contribution | Link |
|---|---|---|
| Finzi et al., 2026 | Epiplexity ($S_T$) — isolating usable structural information from time-bounded entropy | arXiv:2601.xxxxx |
| Su et al., 2026 | EpiSelect & EpiGen — cross-domain scaling laws for data selection; policy gradients over loss-reduction buffers for synthesis | arXiv:2602.xxxxx |
| Information Dilution Theorem | Frames how growing dimensionality degrades metric-information efficiency — relevant for why uniform scaling fails | arXiv:2504.08807 |
For ongoing work, the NeurIPS, ICML, and ICLR proceedings are the primary venues for learning theory, representation learning, and data-centric AI.
6. Closing idea
The cleanest way to think about pre-training data strategy is this: entropy $H$ is the total uncertainty of the source, epiplexity $S_T$ is the structural information extractable within compute budget $T$, and the engineering task is to maximize $S_T$ per FLOP.
Uniform scaling maximizes $H$ while ignoring $S_T$. EpiSelect and EpiGen maximize $S_T$ directly — steering compute toward domains where the loss curve decays fastest, and synthesizing data where natural sources fall short.
Genefold's engineering perspective starts from that first principle: better pre-training is not just more tokens, it is less surprise per compute cycle on the structures that transfer.