Web text
Open-endednats / token
Many valid ways to finish a thought.
Floor 1.69 · α = 0.15
Better models.
Smaller returns.
More compute → better models → diminishing returns → a hard floor.
AI development has a surprisingly predictable shape. Change the scale, and you can see it.
Experiment with the lawA straight line.
An exponential bill.
Training loss measures how surprised a model is by the next token. Lower is better. More compute tends to reduce it by a power law.
On log-log axes, equal distances mean equal multipliers. A pure power law becomes a straight line.
Kaplan et al., 2020 ↗L(C) ∝ C−α
Here α = 0.15, chosen for illustration. The exponent is fitted, not a universal constant.
Loss vs. training computeLOG / LOG
10¹⁸ FLOPs10²⁴ FLOPs
Projected loss nats / token
What this means Every 10× increase buys the same percentage reduction in this model’s loss, not the same absolute decrement. The next decimal point gets progressively more expensive.
One law, with a limit.
The useful extension separates the loss you can reduce from the uncertainty you cannot remove.
L(C) ≈ L∞ + (Cc / C)α
- L∞
- The irreducible floor
- Cc
- A fitted compute scale
- α
- How fast reducible loss falls
Three ways to scale
Parameters NIncrease model capacity, with enough data and training.
Data DIncrease training data, with sufficient model capacity.
Compute CBalance model size and data under a compute budget.
What this means Kaplan et al. found empirical scaling relationships in parameter-limited, data-limited, and compute-optimal regimes. With a nonzero floor, it is excess loss L − L∞, rather than total loss, that stays straight on a log-log plot.
You can approach the floor.
You can’t compute your way through it.
Even infinite data and an ideal, infinitely large model cannot predict genuinely uncertain outcomes. For next-token log loss, that limit is the distribution’s conditional entropy; for classification error, the analogous limit is Bayes error.
Total loss vs. training computeLOG / LOG
Drag vertically in the plot to move the floor, or use the slider.
nats / token
0.302.50
The video framing supplied for this guide cites DeepMind’s estimate of about 1.69 nats per token for general web text. This is a fitted estimate, not a universal constant or a measured hard limit.
Hoffmann et al.’s fit uses E = 1.69. Dataset, tokenizer, and context all matter. See the paper ↗
What this means Extra compute keeps reducing the gap above the floor, while total loss barely moves. Drag the floor to see how changing the data’s uncertainty changes the destination.
Different data. Different horizons.
Structured code and math can be more predictable than open-ended web text. In this illustration, they have lower floors and steeper returns; that ordering is a teaching assumption, not a measured universal ranking.
10¹⁸ FLOPs10²⁶ FLOPs
Code
More constrainednats / token
Syntax and repeated structures constrain choices.
Assumed floor 0.80 · α = 0.22
Math
Highly structurednats / token
Formal relationships can narrow what comes next.
Assumed floor 0.35 · α = 0.30
Entropy and dimensionality are different ideas
Sharma & Kaplan (2020) relate scaling exponents to intrinsic data-manifold dimension under their assumptions: lower dimension can allow faster scaling. They do not establish these three floors. Entropy sets irreducible log loss; intrinsic dimension concerns the structure of the learning problem. Dimension alone does not determine entropy, and predictable text is not necessarily easy to reason about.
What this means Better-curated or more constrained data can change both the learning curve and its limit. Comparing loss across datasets also requires care about tokenization, context, and evaluation setup.
Bigger isn’t always better.
Better fed can be.
A fixed compute budget can buy more parameters or more training tokens. Chinchilla showed that many earlier large models were substantially undertrained: too much model for too little data.
10²¹ FLOPs10²⁶ FLOPs
Optimal parameters · N*
Optimal training tokens · D*
~20 training tokens
per parameter
per parameter
Chinchilla’s 70B parameters and 1.4T tokens motivate this rule of thumb. It is not a universal optimum, especially when inference cost matters.
0.1× · more data10× · less data
Same budget, different allocationsLOG / LINEAR
Your parameters
Your tokens
Loss penalty
What this calculator assumes
For a dense transformer, use training compute C ≈ 6ND and fix the illustrative optimum at D* = 20N*. Then N* = √(C/120). The plotted loss is a symmetric teaching model: L = 1.69 + 0.65(C / (5.88 × 10²³))−0.15(r−0.3 + r0.3), where r = N/N*. This creates a real optimum at r = 1, but is not Hoffmann et al.’s fitted loss function or a forecast for a specific model.
What this means Making the model larger at fixed compute leaves fewer tokens to train it on. In this simplified Chinchilla rule, 4× the compute buys roughly 2× the parameters and 2× the data.
A law of progress.
Or a map of its limits?
The “ideal gas law” reading
A robust empirical regularity can be useful before we fully understand its mechanism. Scaling laws let researchers plan experiments and predict outcomes.
The paradigm-limit reading
The same curves reveal mounting costs and a diminishing return on the current recipe. New data, architectures, or learning methods may matter more at the margin.
The curve tells us what has worked.
Does it tell us what comes next?
What this means Both readings fit the evidence; neither is a proof of what AI must become. Treat scaling laws as empirical guides within tested conditions, and judge the extrapolation for yourself.