Watching Neural Networks Learn

This video explains that neural networks are universal function approximators, learning to model the world by fitting unknown functions from sample data. It walks through the core mechanics—feedforward architectures, weighted sums, activation functions like ReLU and tanh, and backpropagation—before exploring how adding engineered input features can improve learning. Taylor series features prove too finicky, while Fourier features dramatically accelerate and improve approximation in low-dimensional cases like images, though they suffer from the curse of dimensionality in high-dimensional tasks like MNIST. The central lesson is that theoretical elegance isn't enough; practical, empirical testing is essential, and even powerful universal approximators have fundamental limits when confronting functions like the infinitely complex Mandelbrot set.

• ▶ 0:20 Functions describe the world; everything from sound on the eardrum to light hitting the eyes can be modeled as functions.
• ▶ 1:14 The goal of AI is to write programs that understand, model, and predict the world—ideally programs that build their own functions via function approximation.
• ▶ 2:28 Neural networks solve the problem of approximating an unknown function from sample data points by fitting a curve that predicts outputs for unseen inputs, making them universal function approximators.

• ▶ 3:17 The network is a fully connected feedforward function approximating a target, built from neurons that compute weighted sums plus biases, then pass through an activation function like ReLU.

• ▶ 5:03 During training, the weights are adjusted to minimize loss/error via backpropagation, which the narrator promises to explain in a dedicated video.

• ▶ 5:36 The same framework extends to higher-dimensional problems: for image learning, the network maps pixel coordinates (R²) to pixel values (R¹), and slowing learning over time is caused by periodically reducing the learning rate to refine details.

• ▶ 7:12 Input normalization scales raw values (0–1400) to the range -1 to 1 using a linear transformation, which makes learning easier by keeping inputs small and centered at zero.
• ▶ 7:34 Leaky ReLU is used in all layers except the last one because it can output negative values while remaining non-linear, and it generally improves performance compared to standard ReLU.
• ▶ 8:03 The final layer uses tanh instead of sigmoid because tanh outputs values from -1 to 1 and tends to work better with backpropagation, even though the outputs must then be converted to the final 0-to-1 pixel range.
• ▶ 8:36 Rather than relying only on theory, the practical difference is validated empirically by comparing error rates — this “science of math” approach of trying things and measuring results is emphasized as often the most effective method.
• ▶ 9:41 A spiral shell parametric surface is a significantly harder test for the network; despite training on sampled points, the network struggles with the complicated shape, foreshadowing major limits of plain function approximation.
• ▶ 10:22 The Mandelbrot set is introduced as a low-dimensional but infinitely complex function that is uniquely difficult to approximate — no matter how much you fit, the network always comes up short on infinite detail.
• ▶ 13:28 A Taylor series can be framed as a weighted sum of precomputed input features like x, x², x³ — effectively a single-layer neural network — but a first attempt with eight Taylor orders performs poorly, showing that polynomials are "pretty touchy" in practice.

• ▶ 16:15 Fourier features are created by feeding multiple Fourier series terms into a network as additional inputs, and they dramatically outperform both plain networks and Taylor-feature networks in learning speed and accuracy.

• ▶ 17:41 Extending Fourier series to multiple inputs requires a full multidimensional series with multiplied sine/cosine combinations, causing the number of terms to explode—an example of the curse of dimensionality that makes the full approach impractical for high-dimensional inputs.

• ▶ 19:07 A practical workaround is to compute independent 1D Fourier features for each input separately; this is less theoretically rigorous but manageable, and it produces the high-quality image approximations shown earlier.

• ▶ 23:04 Fourier features on MNIST give only ~2% accuracy gain with 8 orders while exploding input size to 13,328 features; 32 orders hurt performance and 64 orders are "downright ruinous."
• ▶ 23:34 The high-dimensional failures are likely due to overfitting, as Fourier networks fail to generalize when model complexity outpaces the data.
• ▶ 24:32 The video closes by challenging viewers to approximate the Mandelbrot set from a random sample, with the rule that the solution must remain a universal function approximator.

• ▶ 0:00 Neural Networks as Functions (0:00 - 3:17) - - Opens with a network learning and recaps functions, curve fitting, and the goal of function building.
• ▶ 3:17 Architecture, Training, and Higher-Dimensional Inputs (3:17 - 7:00) - - Covers network structure, universal approximation, backpropagation training, and learning-rate improvements.
• ▶ 7:00 Practical Improvements and Activations (7:00 - 8:58) - - Discusses input normalization, activation function choices, and evidence-based thinking about networks.
• ▶ 8:58 Harder Problems and Taylor Approximations (8:58 - 15:23) - - Explores parametric surfaces, spiral-shell and Mandelbrot challenges, then Taylor series and Taylor features.
• ▶ 15:23 Fourier Features and Their Impact (15:23 - 21:18) - - Introduces Fourier series and Fourier features, including dimensionality concerns and strong approximation results.
• ▶ 21:18 High-Dimensional Limits and Closing Challenge (21:18 - 25:18) - - Tests Fourier features on high-dimensional MNIST, discusses overfitting, and ends with an open Mandelbrot challenge.