Maximum Likelihood Estimation

1. The Likelihood Function

Let's say we have a bunch of data points $X = \{x_1, x_2, \dots, x_n\}$. We assume these data points come from some probability distribution (like a bell curve) that is controlled by some unknown settings, which we call parameters $\theta$.

The "Likelihood" of those parameters, given the data we saw, is calculated by multiplying the probability of seeing each individual data point:

$\mathcal{L}(\theta | X) = \prod_{i=1}^{n} P(x_i | \theta)$

2. Log-Likelihood

Multiplying a bunch of tiny probabilities together (like $0.01 \times 0.05 \times 0.02$) quickly results in numbers so small that computers round them down to zero. Plus, multiplying is annoying to do calculus on.

To fix this, we take the logarithm of the whole thing. In math, the log of a product becomes the sum of the logs. This turns our multiplication problem into an addition problem, which is much easier for computers to handle:

$\log \mathcal{L}(\theta | X) = \sum_{i=1}^{n} \log P(x_i | \theta)$

3. Finding the Maximum

The Maximum Likelihood Estimate (MLE), written as $\hat{\theta}_{\text{MLE}}$, is simply the parameter value that makes that log-likelihood equation as big as possible:

$\hat{\theta}_{\text{MLE}} = \arg\max_{\theta} \log \mathcal{L}(\theta | X)$

To find this peak, we use calculus. We take the derivative of the log-likelihood equation, set it to zero, and solve for $\theta$:

$\frac{\partial}{\partial \theta} \sum_{i=1}^{n} \log P(x_i | \theta) = 0$

Example: Flipping a Coin

For a coin flip with probability $p$ of getting heads, the math for a single flip is $P(x_i | p) = p^{x_i}(1-p)^{1-x_i}$. The log-likelihood for a bunch of flips is:

$\log \mathcal{L}(p) = \sum_{i=1}^n \left[ x_i \log p + (1-x_i) \log(1-p) \right]$

If you do the calculus (take the derivative and set it to zero), the math perfectly proves that your best guess for $p$ is just the average number of heads you saw: $\hat{p} = \frac{1}{n} \sum_{i=1}^n x_i$.