Skip to main content
Welcome — it’s Alan Chapman from KodeKloud. In this lesson/article we explain how neural networks learn from their mistakes using the chain rule. Backpropagation is the algorithm that fine-tunes a network by reducing prediction error; the chain rule from calculus is the mechanism that makes that possible.
A presentation slide reading "Mathematics for Computing" with the subtitle "Backpropagation and Chain Rule" on a purple dotted background. A woman is standing on the right, speaking or presenting.
Overview
  • Backpropagation: the method for propagating errors backward through a network to update weights.
  • Chain rule: the calculus rule used to compute how each weight affects the final output.
  • Workflow: forward pass → compute loss → backward pass (apply chain rule) → update weights (optimizer).
  • Applications: classification, recommendation systems (e.g., Netflix suggestions), regression, and many other tasks in deep learning.
Quick summary of layer roles: Intuition: Netflix recommendations Imagine you finish watching a series and Netflix starts suggesting shows you will likely enjoy. Early recommendations may feel random; over time the system learns from your choices and adjusts model weights to reduce errors. That adjustment process is backpropagation: it uses gradients computed via the chain rule to move weights in directions that reduce prediction error. Neural network structure (simplified)
  • Inputs: features describing the last show (e.g., genre, setting).
  • One hidden layer (for simplicity) with a single neuron.
  • Output: a prediction score, interpreted as probability.
We keep the diagram simple: two inputs (genre and setting), one hidden neuron, and one output neuron. Forward pass — a concrete numeric example
  • Inputs: genre = 4, setting = 2
  • Initial weights: w_genre = 0.2, w_setting = 0.6
  • Weighted sum into neuron: s = 4 * 0.2 + 2 * 0.6 = 0.8 + 1.2 = 2.0
We use the logistic sigmoid activation so outputs lie in (0, 1) and can be interpreted as probabilities: y = sigmoid(s) = 1 / (1 + exp(-s)) Compute the prediction:
The network predicts about 0.88 (88%). Suppose the actual observed outcome (user interaction) is t = 0.65. The model overestimates the probability, so weights must be adjusted to reduce the prediction. Backpropagation tells us how to change each weight to reduce the loss. Why the chain rule matters The network output y depends on s, and s depends on the weights w. Symbolically: y = f(s(w)). To know how a small change in a weight w changes y, we compute dy/dw using the chain rule: dy/dw = (dy/ds) * (ds/dw)
  • dy/ds is the derivative of the activation function with respect to its input s.
  • ds/dw is the derivative of the weighted sum with respect to the weight w (usually the corresponding input value).
Sigmoid derivative and example
  • For sigmoid: dy/ds = sigmoid(s) * (1 - sigmoid(s)).
  • For s = 4 * w_genre + 2 * w_setting, ds/dw_genre = 4.
Evaluate these numerically for our example:
Gradient of the loss w.r.t. a weight If we use squared error loss E = 1/2 * (y - t)^2, then: dE/dy = (y - t) and by the chain rule: dE/dw = (dE/dy) * (dy/dw) For t = 0.65:
A single gradient-descent update (learning rate lr = 0.1):
Interpretation
  • The update moves the weight opposite to the gradient (hence the minus sign) to reduce loss.
  • Apply the same process to every weight (e.g., w_setting) and update them together.
  • Repeating over many examples and epochs is how backpropagation trains the network.
Compact conceptual summary We used the chain rule to express how a small change in a weight affects the final error, then used that gradient with an optimizer to update the weight to reduce error. Composite functions and the chain rule Composite functions are “functions inside functions” — the mathematical equivalent of stacked layers in neural networks. Example: y(x) = 3 x^2 + 1
z(y) = y^3 + 3
Combined: z(x) = (3 x^2 + 1)^3 + 3 To compute dz/dx: dz/dx = dz/dy * dy/dx
  • dy/dx = 6 x
  • dz/dy = 3 y^2
So: dz/dx = 3 y^2 * 6 x = 18 x y^2 = 18 x (3 x^2 + 1)^2
A presentation slide titled "Composite Functions" with equations, derivative notation, and a purple 3D cube in the center. A presenter is visible on the right side of the image, speaking and gesturing.
Bakery analogy (another composite example) Think of a bakery where:
  • y(x) = x^3 - 2 # dough quality depends on yeast x
  • z(y) = 2 - y^2 # bread quality depends on dough quality
Combined: z(x) = 2 - (x^3 - 2)^2 Apply the chain rule:
  • dy/dx = 3 x^2
  • dz/dy = -2 y
So dz/dx = dz/dy * dy/dx = (-2 y) * (3 x^2) = -6 x^2 y = -6 x^2 (x^3 - 2) Evaluate at x = 1:
  • y = 1^3 - 2 = -1
  • dz/dx = -6 * 1^2 * (-1) = 6
A positive derivative here means increasing yeast slightly (near x = 1) improves bread quality; derivatives identify the direction and rate of change and help find optima.
A dark slide showing a glowing purple curve on a graph labeled "x (Amount of yeast)" vs "z (Quality of bread)". A smiling presenter stands at the right, gesturing with her hands.
Back to neural networks — putting it together
  • Forward pass: compute s = sum(inputs * weights) and y = activation(s).
  • Backward pass: compute derivatives using the chain rule: dE/dw = (dE/dy) * (dy/ds) * (ds/dw)
Each factor is computed locally (at the neuron or layer) and multiplied together — that is the chain rule applied across layers. A helpful mapping of common derivative components: After computing dE/dw for every weight, the optimizer (gradient descent, SGD, Adam, etc.) adjusts weights accordingly. A visual recap: the chain rule is applied layer-by-layer to propagate error derivatives back to earlier weights, enabling each weight to be updated according to its contribution to the error.
A presentation slide titled "Neural Network for Netflix Recommendation" showing derivative math, a purple 3D cube graphic, and a presenter gesturing on the right. The background is dark purple/black with stylized dots on the left.
Practical notes
Backpropagation uses the chain rule to compute gradients efficiently. In practice, modern frameworks like TensorFlow and PyTorch compute these derivatives automatically via autodiff, but understanding the chain-rule flow is essential for debugging, optimizing architectures, and designing custom layers or loss functions.
Conclusion
  • Composite functions are the mathematical form of layered neural networks.
  • The chain rule multiplies derivatives of inner and outer functions to determine how small changes propagate.
  • Backpropagation repeatedly applies the chain rule across layers to compute gradients and update weights, enabling networks to learn from data.
A dark-themed presentation slide titled "Conclusion" listing short summaries of Composite Functions, the Chain Rule, and Backpropagation. A woman stands on the right, gesturing with her hands as she speaks.
Further reading and references This was a simplified demonstration showing how backpropagation and the chain rule work together to adjust network weights and improve predictions over time.

Watch Video