Backpropagation calculus | Deep Learning Chapter 4

This video explains how machine learning practitioners apply the chain rule to neural networks, shifting from the typical calculus textbook view to a practical sensitivity-focused approach. Using a minimal network model, it shows how to compute the derivative of the cost with respect to each weight and bias by breaking the change into manageable factors: how a weight affects the neuron's weighted input, how that input affects the activation, and how the activation affects the cost. The key insight is that a weight’s influence is scaled by the previous layer’s activation, illustrating the principle that "neurons that fire together, wire together." The full cost derivative is simply the average of per-example derivatives, and this gradient component guides the network toward minimizing error. By tracking sensitivity backward through layers, the same chain-rule logic can be applied iteratively—this is the core of backpropagation. Extending to multiple neurons per layer only adds indices, not new concepts, since each earlier neuron's total influence is the sum of its contributions through multiple paths. Ultimately, these expressions form the gradient that lets the network step downhill to reduce cost.

• ▶ 0:22 The main goal is to show how machine learning practitioners think about the chain rule in neural networks, which differs from the typical intro-calculus presentation.
• ▶ 0:40 The example uses an extremely simple network with one neuron per layer, defined by three weights and three biases, to study how sensitive the cost function is to each variable.
• ▶ 1:24 For one training example, the cost is C₀ = (a^(L) − y)², and the chain of dependencies runs from weight, previous activation, and bias → z → a → cost.
• ▶ 2:42 Goal is to compute the sensitivity of the cost to the final weight: ∂C/∂w(L), measured by nudging the weight and observing the change in cost.
• ▶ 3:11 The chain rule breaks this into three factors: ∂z(L)/∂w(L), ∂a(L)/∂z(L), and ∂C/∂a(L), which multiply to give ∂C/∂w(L).
• ▶ 4:46 Intuition: the weight's influence is scaled by the previous activation a(L-1), embodying "neurons that fire together, wire together"; these derivatives describe sensitivity for a single training example.
• ▶ 5:16 The full cost function derivative is the average of per-example derivatives, and this averaged derivative is only one component of the full gradient vector.
• ▶ 5:58 The bias sensitivity derivative simplifies to 1, making it especially simple compared to the weight sensitivity.
• ▶ 6:34 By tracking sensitivity to previous activations, the same chain rule can be applied iteratively backwards, layer by layer, to find sensitivities for all previous weights and biases.
• ▶ 6:43 Moving to multiple neurons per layer may seem more complex, but not much actually changes — it mostly requires a few more indices to keep track of activations, costs, and weights.
• ▶ 7:07 New notation uses k for neurons in layer L-1 and j for neurons in layer L; the cost sums squared differences over all output neurons, and weights are labeled w(L)_jk for the edge from neuron k in layer L-1 to neuron j in layer L.
• ▶ 8:15 The chain-rule expression for how sensitive the cost is to a specific weight remains essentially the same as the one-neuron case, just with added indices — viewers are encouraged to pause and work through each term.
• ▶ 8:38 A neuron in an earlier layer affects the cost through multiple paths, so its overall influence is the sum of all those path contributions.
• ▶ 9:00 Once sensitivities for one layer are known, the same process is repeated backward for earlier weights and biases — this is the core mechanism of backpropagation.
• ▶ 9:14 These chain rule expressions give the gradient, which lets the network minimize cost by repeatedly stepping downhill.

• ▶ 0:00 Introduction, Network Setup, and Notation (0:00 - 2:42) - Introduces the simple network, notation, and cost definition.
• ▶ 2:42 Chain Rule and Weight Sensitivity (2:42 - 5:16) - Uses the chain rule to compute the cost derivative with respect to the last weight and activation.
• ▶ 5:16 Gradient Averaging and Bias Sensitivity (5:16 - 6:43) - Averages derivatives over training examples and finds bias sensitivity analogously.
• ▶ 6:43 Generalizing Backpropagation to Multiple Neurons (6:43 - 8:29) - Extends backpropagation to real networks with multiple neurons per layer.
• ▶ 8:29 Multiple Paths and the Heart of Backpropagation (8:29 - 9:43) - Sums contributions from multiple paths and repeats the process backward through the network.