Calculus & Optimisation

1. The Derivative

A derivative simply measures how fast something is changing at a specific moment. Mathematically, it looks like this:

$f'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h}$

In machine learning, if $f(x)$ is our error (or loss), the derivative $f'(x)$ tells us exactly how much our error will go up or down if we tweak our parameter $x$ just a tiny bit.

2. Partial Derivatives and the Gradient

When we have lots of parameters $\mathbf{w} = [w_1, w_2, ..., w_n]$, we calculate the slope for each one individually while pretending the others are frozen. When we put all these individual slopes together into a list, we call it the Gradient, $\nabla L$:

$\nabla L(\mathbf{w}) = \begin{bmatrix} \frac{\partial L}{\partial w_1} \\ \frac{\partial L}{\partial w_2} \\ \vdots \\ \frac{\partial L}{\partial w_n} \end{bmatrix}$

To improve the model, we subtract a small fraction of this gradient from our current weights. The size of the step we take is controlled by a setting called the learning rate, $\alpha$:

$\mathbf{w}_{new} = \mathbf{w}_{old} - \alpha \nabla L(\mathbf{w}_{old})$

3. The Chain Rule

The Chain Rule is the true hero of deep learning. It tells us how to find the derivative of functions that are stuffed inside other functions. If $y = f(u)$ and $u = g(x)$, the rule says:

$\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}$

Neural networks are basically just giant chains of functions: $Output = f_3(f_2(f_1(X)))$. The chain rule lets us work backwards from the final error, multiplying slopes together to figure out exactly how to adjust every single weight in the network.