Free AI summary of this YouTube video by SumTube
But what is cross-entropy? | Compression is Intelligence Part 2
TL;DR
This video explores cross-entropy, bridging its origins in data compression with its modern role as the primary loss function for training large language models (LLMs). Grant Sanderson demonstrates how compression algorithms like gzip measure linguistic similarity and mathematically proves why cross-entropy naturally emerges when optimizing next-token predictions. The lesson connects information theory, gradient descent, model distillation, and KL divergence to show how LLM training is fundamentally about text compression.
Key takeaways
- 8:01Cross-entropy measures the inefficiency of using a compression scheme optimized for one probability distribution on data generated by another distribution
- 41:30Logarithmic functions are uniquely forced by calculus (via Lagrange multipliers) as the only loss functions that minimize average loss when predicted probabilities match true data frequencies
- 31:56KL divergence quantifies the exact number of wasted bits per symbol when using an imperfect model distribution compared to the true distribution

Tap to enlarge
Most valuable section
For whom
Engineers and students wanting an intuitive, visual mathematical proof of why cross-entropy loss works in deep learning.
Mathematical proof for logarithmic loss23:38
It uses visual graphs and multivariable calculus to prove why logarithmic functions are strictly required to align model predictions with data statistics.
Detailed summary
1. Language Trees and Zipping
0:00The video opens by examining a 2002 research paper titled 'Language Trees and Zipping' which demonstrated that basic file compression tools like gzip can cluster text documents by language and reconstruct evolutionary language trees without any prior linguistic knowledge. The technique works by appending a small text snippet from one document to another and compressing the combined file. By comparing the size of the combined compressed file against the compression size of the original document alone, researchers obtained a distance metric based strictly on shared structural patterns.
This compression-based distance metric provides a pure example of how mathematical compression theory solves problems typically reserved for artificial intelligence and machine learning. The core concept driving this phenomenon is cross-entropy, a fundamental principle in information theory that also governs how modern large language models are pre-trained. By returning to first principles, this lesson sets out to connect file compression efficiency with the loss functions used in deep learning.
2. Recap: Information Theory Basics
5:02To build an intuitive foundation for cross-entropy, the video recaps key principles of information theory using a rover instruction toy example. When transmitting messages composed of discrete symbols sampled from a probability distribution, optimal encoding schemes assign shorter bit-lengths to high-frequency symbols and longer bit-lengths to rare symbols. Claude Shannon formally defined the information content of an event as negative log base 2 of its probability, representing the exact theoretical number of bits required to encode that symbol.
While individual symbol bit-lengths are often fractional values in practice, encoding full messages allows these fractional information values to sum up to the total bit length of an optimally compressed file. The theoretical minimum average bit-length per symbol for a given probability distribution is called its entropy. Understanding optimal encoding length as a function of event probability provides the exact framework needed to analyze what happens when an encoding scheme is misaligned with the real world.
3. Defining Cross-Entropy
8:41The video introduces cross-entropy by altering the rover example so that the frequency of command symbols changes, while the transmission system remains hardcoded to the old optimal encoding scheme. Calculating the new average bits required per message reveals an increased value due to coding inefficiency. Mathematically, cross-entropy is the weighted sum of the bit-lengths from an old or assumed distribution Q using the actual probabilities of a new distribution P.
Visualized on a bar chart, cross-entropy represents the total area formed when bar widths correspond to true probabilities P and bar heights correspond to the negative log-probabilities of Q. Exploring simple two-event probability distributions demonstrates that cross-entropy is asymmetric; swapping the roles of P and Q yields different average bit lengths depending on which distribution was used to design the code versus which distribution actually occurred.
4. Key Property of Cross-Entropy
17:30By plotting cross-entropy as a function of predicted probabilities Q while holding the true distribution P fixed, a critical mathematical property emerges: the cross-entropy curve reaches an absolute minimum exactly when distribution Q equals distribution P. At this minimum point, the value of cross-entropy equals the theoretical entropy of distribution P. This means any deviation between predicted probabilities and true frequencies strictly increases average bit-length.
This core property explains why cross-entropy is directly applicable to language model training. In language clustering tasks, such as the 'Language Trees and Zipping' paper, appending text snippets measures how well a compressor optimized for document A compresses document B. The resulting size difference serves as an empirical estimate of cross-entropy, quantifying how closely the linguistic patterns of the two texts align.
5. Language Model Pre-training & Loss Functions
24:59The focus shifts to modern machine learning, recapping how large language models break text into tokens and output probability distributions predicting the next token. Training an LLM requires a loss function—a scalar value measuring prediction quality—so gradient descent and backpropagation can iteratively adjust model parameters to minimize loss. While visualizing a loss function as a 3D terrain helps build intuition, real LLM loss landscapes operate across hundreds of billions of parameter dimensions.
For a language model, loss is evaluated against massive web text datasets. A natural framing for loss is measuring average information content from the model's perspective across every token in the dataset. A well-trained model assigns high probability to true next tokens, experiencing low surprise (low loss), whereas an untrained model assigns low probability to correct tokens, resulting in high surprise and steep penalty values.
6. Why Cross-Entropy Loss Uses Logarithms
31:46While many mathematical functions penalize low probabilities with high values, logarithmic loss is uniquely required for machine learning. When evaluating a model over repeated dataset prompts (such as 'My name is [blank]'), total average loss across all dataset instances forms a weighted sum of the loss function applied to predicted probabilities. Using multivariable calculus and Lagrange multipliers, the video proves that a logarithmic curve is the only function that guarantees average loss is minimized if and only if predicted probabilities match true data frequencies.
In machine learning, natural logarithms are preferred over log base 2 because their derivative (1/x) simplifies gradient calculations during backpropagation. Thus, 'cross-entropy loss' is not merely an arbitrary choice; calculus forces its log-based structure to ensure that optimizing loss directly drives the model's internal probability estimates to match real-world data statistics.
7. Model Distillation
35:33The video examines model distillation, an advanced training variant where cross-entropy loss is applied explicitly across full probability distributions. In distillation, a small, efficient student model is trained to mimic a much larger, smarter teacher model. Instead of evaluating the student model against a single true next token from raw text, the student's output probability distribution is compared directly against the teacher's full predicted probability distribution across all possible tokens.
Evaluating loss using cross-entropy between the student and teacher distributions provides richer training signals than standard pre-training. An analogy is drawn to learning chess: standard pre-training is like learning chess by observing individual move outcomes, whereas distillation is like receiving detailed coaching on the relative quality of every candidate move at every turn. Distillation enables smaller models to rapidly internalize complex reasoning patterns developed by larger architectures.
8. KL Divergence
37:12The video concludes by introducing Kullback-Leibler (KL) divergence, a closely related concept widely used in machine learning papers. KL divergence is defined as cross-entropy minus the theoretical entropy of the true distribution. In information theory terms, KL divergence measures the exact number of extra or 'wasted' bits per symbol incurred by using an inaccurate probability model Q instead of the true distribution P.
Because true data entropy is constant with respect to model parameters, minimizing KL divergence during neural network training is mathematically identical to minimizing cross-entropy loss. Three concluding thought exercises encourage viewers to verify algebraic equivalences of KL divergence formulas, interpret its visual area diagrams, and analyze how replacing cross-entropy with KL divergence impacts model distillation performance.
Questions this video answers
What is cross-entropy in the context of machine learning?
In machine learning, cross-entropy serves as a loss function that quantifies the difference between a model's predicted probability distribution and the actual distribution of training data. Minimizing cross-entropy encourages the model to assign higher probabilities to true target outcomes, effectively improving prediction accuracy.
Why do neural networks use logarithms in loss functions instead of other curves?
Using Lagrange multipliers, calculus proves that logarithmic functions are the only curves where average loss is minimized if and only if the model's predicted probabilities perfectly match the underlying data frequencies. Additionally, natural logs yield clean derivatives (1/x) which streamline gradient descent optimization during training.
How is language model pre-training related to data compression?
Pre-training an LLM minimizes the negative log-probability of true target tokens across billions of context examples. According to information theory, this negative log-probability represents the optimal bit-length required to encode a symbol, meaning an LLM trained on cross-entropy loss is literally learning to compress language at Shannon's theoretical limit.
What is the difference between cross-entropy and KL divergence?
Cross-entropy measures the total average information (or bits) required to encode events from a true distribution P using a code optimized for distribution Q. KL divergence is the difference between cross-entropy and the true distribution's theoretical minimum entropy, specifically measuring the extra or 'wasted' bits resulting from using Q instead of P.
Ask a follow-up Premium
About this summary
SumTube AI wrote this recap from the video's words and chapters. Credit and the original belong to 3Blue1Brown.
