But what is cross-entropy? | Compression is Intelligence Part 2

This video explains how cross-entropy—the core objective behind modern language model training—can be understood through the lens of data compression, beginning with a striking 2002 experiment where gzip alone reconstructed language family trees. It introduces a toy example of robot instructions to show that optimal codes assign bits according to negative log probabilities, and that when the true distribution shifts, using an outdated code incurs an average cost known as cross-entropy. The key insight is that cross-entropy measures how well a code designed for one distribution encodes another, and it is minimized—but never reduced to zero—when the encoding distribution exactly matches the true distribution, with the minimum equal to the entropy itself. This principle makes cross-entropy a practical way to quantify the gap between a language model's learned patterns and the true patterns in training data, connecting abstract information theory directly to LLM training.

• ▶ 0:00 A 2002 paper showed that standard file compression (gzip) can cluster languages and reconstruct their evolutionary tree using no linguistic knowledge.
• ▶ 0:51 The method compares compressed sizes: append a snippet of document B to A, compress, and measure the size increase to define a co-compression-based distance between documents.
• ▶ 1:43 This works because of cross-entropy, the same concept central to training modern language models—framing LLM training not as next-token prediction but as compression.
• ▶ 3:02 Cross-entropy is motivated through encoding messages as sequences of bits, building on the framing of messages as symbol sequences.
• ▶ 3:14 Symbols are sampled from a probability distribution, and those probabilities set fundamental limits on how much a message can be compressed.
• ▶ 3:31 Because natural language is too complex, a simplified toy example is used—and that same example is what motivates cross-entropy itself.
• ▶ 3:39 The video sets up a concrete toy example: four robot instructions (up, down, left, right) with probabilities 1/2, 1/4, 1/8, and 1/8.
• ▶ 4:15 The key general principle: in an optimal code, the number of bits for a symbol is negative log base 2 of its probability (e.g., 1 bit for a 1/2-probability event, 2 bits for 1/4, 3 bits for 1/8).
• ▶ 4:45 Shannon defined this expression as the information content of an event; even fractional information values are meaningful, and the total information of a message is the sum of its individual symbols' information.
• ▶ 5:21 When the true symbol distribution changes (up/down each 1/8, left 1/4, right 1/2), the old encoding system remains hard-coded to the previous scheme, causing inefficiency.
• ▶ 5:46 The exact cost of this mismatch can be computed as a weighted sum: (1/8)(1) + (1/8)(2) + (3/4)(3) = 2.625 bits per message.
• ▶ 6:14 This average cost is the cross-entropy of the original distribution relative to the new one, measuring how well an encoding optimized for one context performs in another.
• ▶ 6:41 Define (P) as the actual/probability distribution of messages you receive, and (Q) as the distribution the encoding was originally designed for.
• ▶ 6:49 The logic generalizes beyond four robot instructions to any sequence of symbols, such as English text.
• ▶ 7:03 The average number of bits per symbol is (-\sum_i P_i \log_2 Q_i), where (P) provides the weights and (Q) provides the code lengths.
• ▶ 7:27 Entropy of Q is visualized as a diagram where bar width = Q and bar height = -log₂(Q), making each weighted term's contribution explicit.
• ▶ 7:43 When the actual distribution is P instead of Q, the average bits per instruction is still a weighted sum, but the weights become the true probabilities P, not the encoding-optimized Q.
• ▶ 8:00 This generalized sum is defined as cross-entropy of Q relative to P; the updated diagram uses bar widths = Pᵢ and bar heights = -log₂(Qᵢ), mixing the actual distribution with the encoding distribution.
• ▶ 8:26 With two events, cross-entropy is first demonstrated using an even Q (50/50) and a skewed P (90/10).
• ▶ 9:57 Reversing the roles, using a code optimized for a skewed distribution on equally likely events gives a cross-entropy of ~1.74 bits, showing a clear disadvantage.
• ▶ 10:11 Order matters: swapping P and Q changes cross-entropy because the two distributions play fundamentally different roles in the formula.
• ▶ 10:22 With only two outcomes, both Q and P reduce to a single free parameter, making the cross-entropy expression directly graphable.
• ▶ 10:51 Fixing P and plotting cross entropy as a function of Q reveals a clear minimum exactly at P = Q, matching the intuition that compression is most efficient when the code's setting matches reality.
• ▶ 11:28 Moving P changes the graph and traces out a green curve showing the minimum cross-entropy value, which is precisely the entropy H(P) as a function of p_1.
• ▶ 12:09 Cross-entropy reaches its minimum exactly when Q equals P, and that minimum value is the entropy of P, not zero.
• ▶ 12:25 The property is easy to see with two outcomes but hard to visualize generally; the video anchors it with a bar diagram where P's widths are fixed and Q determines the bar heights (negative log base 2 values).
• ▶ 12:49 Core lesson: cross-entropy measures how well Q encodes P, and it can never go below the entropy of P — it only matches it when Q equals P.

• ▶ 13:14 The core idea is to measure text difference by appending a snippet of document B to document A, compressing the combined file, and comparing its size to compressing A alone.

• ▶ 13:31 The size difference reflects how well a compression scheme optimized for one context handles another context—the same question cross-entropy asks, though here it is an empirical estimate rather than a direct comparison of probability distributions.

• ▶ 14:26 Even though gzip is far from perfect compression, the authors’ approximation of cross-entropy still yields a useful distance measure, enabling tasks like comparing languages and building language trees.

• ▶ 14:41 Cross-entropy is commonly used in practice to quantify how different the patterns in one setting are from those in another, such as comparing the patterns of two languages.
• ▶ 14:52 The video transitions to the main application: measuring how different a language model's understanding of language is from the true patterns of language.
• ▶ 15:01 In practice, the "true" patterns are represented by training data, making cross-entropy a practical tool for measuring the gap between what the model has learned and what the data shows.
• ▶ 15:10 The animation seen throughout the discussion represents the underlying transformer architecture, created by a friend of the narrator.
• ▶ 15:16 The video will now shift to a more schematic "black box" representation of the transformer.
• ▶ 15:20 This black box is controlled by a large number of tunable parameters, abstracting the learned weights of the neural network.
• ▶ 15:36 Language models split text into tokens (words or word pieces), not raw characters or sentences.
• ▶ 15:41 A language model is a function that takes any piece of text as a sequence of tokens as input.
• ▶ 15:48 Its output is a probability distribution over all possible tokens, representing the prediction of what comes next.
• ▶ 15:59 A loss function quantifies whether the model's predictions are generally good or bad, providing the numerical signal for training.
• ▶ 16:08 The loss is often visualized as a surface to hunt for the minimum, but this picture must be taken with a "heavy grain of salt."
• ▶ 16:20 The real loss function has billions of parameters (dimensions), yet the simplified "descending a hill" idea captures the essence of automatically tweaking parameters to decrease the loss (▶ 16:34–▶ 16:40).
• ▶ 16:46 Gradient descent and backpropagation handle the automatic tuning of model parameters, representing the standard machinery of deep learning.
• ▶ 16:54 From the engineer's perspective, the main task is simply to define a good loss function; once that is done, the optimization machinery takes over.
• ▶ 17:15 The central design question is crafting a mathematical loss function over the training data such that minimizing it is exactly equivalent to improving the model.
• ▶ 17:28 Loss is the average information per token from the model's perspective.
• ▶ 17:38 The model predicts each next token from every initial subsequence, and the loss is based on the probability assigned to the true next token.
• ▶ 18:17 The loss takes the negative log of those probabilities, measuring surprise—a smart model is unsurprised, while a poor model is constantly surprised.

• ▶ 18:57 The loss function assigns a high penalty when the true next token has low probability, and only a tiny penalty when it has high probability.

• ▶ 19:05 Many functions share this same qualitative shape, so the section asks why information/log-based loss is specifically the best choice.

• ▶ 19:17 The key open question is what makes logs special among all plausible loss functions with that shape, motivating a justification for cross-entropy.

• ▶ 19:21 In machine learning, the natural log (base e) is used instead of log base 2, though the difference is only a constant factor and doesn't affect model behavior.
• ▶ 19:30 This constant factor is absorbed by the learning rate, so the optimization process is unaffected by the choice of log base.
• ▶ 19:33 Natural logs are preferred for mathematical convenience because they are cleaner to differentiate, which simplifies gradient descent for decreasing the loss.
• ▶ 19:46 Pre-training is conceptually simple: it's just one repeated operation across a massive dataset.
• ▶ 19:48 The core mechanism is averaging the negative log probability the model assigns to each token over all tokens seen.
• ▶ 19:58 This averaging applies to every token in the entire training set, which is treated as a representative sample of the language.
• ▶ 20:08 Practical training involves more than the loss itself, including batching, specific optimizers, and large-scale engineering feats.
• ▶ 20:19 At the loss-function abstraction level, things are remarkably simple: the loss is just the average information per token.
• ▶ 20:30 Minimizing this loss should bring it close to the entropy of language, resulting in a powerful general predictor.
• ▶ 20:38 The loss function is surprisingly named "cross-entropy loss," not "log loss" or "information loss," despite no cross-entropy formula being used so far.
• ▶ 20:53 The textbook explanation: cross-entropy is computed between the model's output distribution and a skewed distribution placing all probability on the true next token, which reduces to the negative log probability of that token.
• ▶ 21:23 The narrator calls this explanation "very lame" because it doesn't justify why cross-entropy is the right concept—the formula just collapses into simple negative log probability.
• ▶ 21:52 A loss function must be decreasing in the probability of the correct token, but infinitely many functions satisfy this; an additional property is needed to choose one.
• ▶ 23:35 Averaging the loss over all examples of a pattern yields a weighted average of the loss F(Q) weighted by the data frequencies P, which naturally leads to cross-entropy.
• ▶ 24:30 The log is forced: demanding that the average loss be minimized only when the model’s output distribution equals the data statistics mathematically forces F to be the negative logarithm (via a Lagrange multiplier argument).
• ▶ 26:17 Distillation is a training setup that uses cross-entropy much more explicitly than standard pre-training: each example gives a softer, more distributed tug on the model's distribution rather than over-weighting one specific next token.
• ▶ 26:32 The goal is to train a relatively small model to approximate the behavior of a larger, smarter model—so you can scale up for quality, then distill down for efficient inference.
• ▶ 27:02 At each token, the small model's full prediction distribution is compared to the larger model's full distribution (not just the true next token), producing a much richer training signal from each example.
• ▶ 28:15 Cross-entropy connects the two halves of the discussion: it arises naturally in compression as a measure of pattern mismatch, and also appears in constrained optimization as a loss function—the recurring link is fundamentally tied to compression.
• ▶ 28:57 Like pi in geometry, cross-entropy appears in seemingly unrelated contexts, but there is always a hidden connection to compression; for language models, the loss function is already described as the average information per token from the model's perspective.
• ▶ 29:22 The next key step is turning a predictor (a language model) into a compressor, so that the bits used for text match its model-perspective information content—leading to the equivalence: cross-entropy loss is training the model to be the best possible text compressor, directly testing "compression is intelligence."
• ▶ 30:12 The speaker introduces a footnote covering KL divergence, a concept frequently seen in machine learning papers.
• ▶ 30:24 A plug is made for the 3b1b talent page, which hosts a puzzle directly related to KL divergence as a teaser.
• ▶ 30:36 The partner page at 3b1b.co/talent features organizations keen to hire the video's audience, with interviews to convey company culture better than a generic jobs page.
• ▶ 30:59 Many companies offer featured challenges designed for viewers, and applicants are invited to mention their solutions in job applications.
• ▶ 31:09 By coincidence, Dopple posted a challenge using KL divergence, providing real-world evidence that the video's math appears outside academia — unplanned yet fitting.
• ▶ 31:38 KL divergence is defined as the gap between cross-entropy and its minimum value (the entropy of p), which occurs when the predicted distribution equals the true distribution; this gap is named the Kullback-Leibler divergence ▶ 31:55.
• ▶ 32:02 In compression, KL divergence measures how many bits per symbol are wasted by a poorly optimized code; in machine learning, it acts as a kind of distance measure between distributions, being zero when identical and growing as they differ ▶ 32:25.
• ▶ 32:34 KL divergence is not a true distance metric because it is asymmetric: the KL divergence of q relative to p is not the same as the KL divergence of p relative to q.
• ▶ 32:47 The cross-entropy expression is often written in a compact form; verify that it is equivalent to cross-entropy minus entropy.
• ▶ 33:02 Pause and connect the repeated diagram to the mathematical expression, treating it as a visual variant of cross-entropy.
• ▶ 33:11 Consider why distillation uses cross-entropy rather than KL divergence, and what would change if you swapped in KL divergence.

• ▶ 0:00 1. Compression, Zipping, and Cross-Entropy (0:00 - 3:02) - Introduces the zipping paper and the plan to use cross-entropy as the key lens.
• ▶ 3:02 2. Cross-Entropy from First Principles (3:02 - 13:00) - Motivates cross-entropy through encoding, information content, and two-outcome visualizations.
• ▶ 13:00 3. Compression Distance as Text Similarity (13:00 - 14:41) - Shows how zipping approximates cross-entropy for comparing languages and texts.
• ▶ 14:41 4. From Practice to Language-Model Loss (14:41 - 21:38) - Moves from cross-entropy in practice to next-token prediction, loss functions, and pre-training.
• ▶ 21:38 5. Deriving the Logarithmic Loss (21:38 - 26:17) - Justifies why the loss must be a logarithm and how cross-entropy emerges from weighted averages.
• ▶ 26:17 6. Distillation and Recap (26:17 - 33:28) - Covers explicit cross-entropy distillation and recaps the links between compression, cross-entropy, and language modeling.