ML Learn

Learning AI & ML

Logistic Regression

Logistic Regression

Difficulty:Intermediate
Reading Time:25 min
Track:
Practitioner
A classification model that uses a sigmoid function to map a linear combination of features to a probability between 0 and 1.

Prerequisites

Linear Regression
ML PractitionerModule 3 of 17

Logistic Regression

TL;DR

  • Logistic regression predicts a probability by squashing the linear score z=wTx+bz = w^T x + b through the sigmoid σ(z)=1/(1+e−z)∈(0,1)\sigma(z) = 1/(1 + e^{-z}) \in (0, 1).
  • It is trained by minimizing binary cross-entropy (log-loss), which is the negative log-likelihood of the data under a Bernoulli model.
  • Log-loss is convex in the parameters, while MSE-of-sigmoid is not — so cross-entropy gives a clean, single global optimum.
  • Coefficients are interpretable as log-odds: a one-unit increase in feature xjx_j multiplies the odds of the positive class by ewje^{w_j}.
  • The gradient simplifies to 1nXT(p^−y)\frac{1}{n}X^T(\hat{p} - y) — the same "prediction-minus-target" form as linear regression — and there is no closed form, so it is fit iteratively.
  • You turn a probability into a class by comparing it to a threshold (default 0.50.5), which you can tune to trade off precision and recall.

Learning Objectives

  1. Map linear scores to probabilities using the Sigmoid (logistic) function
  2. Formulate the Binary Cross-Entropy loss function and explain its derivation from MLE
  3. Compute prediction values and evaluate log-loss for classification model outputs
  4. Interpret model coefficients as changes in log-odds

Intuition

How to think conceptually about this topic

Logistic regression is the boundary problem. The model calculates a linear score for each data point based on its features. This score indicates which side of a decision boundary the point falls on. The sigmoid function then converts the distance from this boundary into a probability: points far on one side have a probability near 1, points far on the other side have a probability near 0, and the decision boundary itself is the line of maximum uncertainty where the probability is exactly 0.5.

Interactive Diagram

Test the intuition above by changing the model parameters

In Depth

Detailed explanations, contexts, and details

Logistic regression is a fundamental classification algorithm in machine learning. Despite the word "regression" in its name, it is used for binary classification tasks. The key idea is to take the same linear combination of features used in linear regression, but pass the output score through a non-linear activation function—the Sigmoid—to map it to a probability value between 0 and 1.

Pedagogically, logistic regression demonstrates how we extend linear models to categorical outputs, why squared error is unsuitable for classification, and how cross-entropy loss and maximum likelihood estimation guide the optimization process.

Where is it used?

Logistic regression is widely used for binary classification tasks where probability estimates and interpretability are important, such as user churn prediction (churn/no churn), fraud detection (fraud/legitimate), clinical diagnostics (disease present/absent), or email spam filtering.

How It Compares

Logistic Regression vs Linear Regression vs SVM (classification)

DimensionLogistic RegressionLinear RegressionSVM (classification)
Output typeProbability in (0,1)(0, 1) via sigmoidUnbounded continuous value (not a probability)Signed distance / class label (margin score)
Loss functionBinary cross-entropy (log-loss)Mean squared errorHinge loss (max-margin)
Decision boundaryLinear (hyperplane where z=0z = 0)Not designed for boundaries; thresholding a line is ad hocLinear, or non-linear via kernels
Probability estimatesYes — native, well-calibratedNoNo native probabilities (needs Platt scaling)
InterpretabilityHigh — coefficients are log-oddsHigh — coefficients are slopesLower — especially with kernels
TakeawayFor binary classification with interpretable probabilities, reach for logistic regression. Linear regression is the wrong tool for classification (it has no probability or boundary semantics), and SVMs maximize the margin but do not give probabilities out of the box.

When to Use It

Reach for this when

  • You need calibrated probability estimates, not just hard class labels (e.g. risk scores, ranking by likelihood).
  • You need an interpretable classifier whose coefficients (as log-odds / odds ratios) you can explain to stakeholders or regulators.
  • You want a fast, low-variance baseline for a binary (or, via softmax, multiclass) classification problem.

Avoid it when

  • The decision boundary is strongly non-linear and resists feature engineering — prefer trees, gradient boosting, or neural networks.
  • Classes are perfectly (or near-perfectly) separable, which causes weights to diverge — regularize, or use a margin method.
  • Features are highly collinear or the dataset is wide (p≫np \gg n) without regularization — coefficients become unstable.

Rules of thumb

  • Standardize features so coefficient magnitudes are comparable and L1/L2 regularization behaves sensibly.
  • Always apply some regularization (L2 by default) to guard against complete separation and overfitting.
  • Tune the decision threshold using a precision-recall or ROC analysis — do not blindly keep 0.50.5 when class costs are asymmetric.

Implementation

Reference code implementation

Python
model_fitting.py
1import numpy as np
2from sklearn.linear_model import LogisticRegression
3from sklearn.metrics import log_loss
4
5# Features: [hours studied, practice-test average]
6X = np.array([
7    [1.2, 2.0],
8    [2.0, 3.2],
9    [2.8, 2.4],
10    [3.6, 4.0],
11    [5.2, 5.1],
12    [6.4, 5.8],
13    [7.1, 7.4],
14    [8.2, 6.8],
15    [8.8, 8.5],
16])
17y = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1])
18
19clf = LogisticRegression()
20clf.fit(X, y)
21prob = clf.predict_proba(X)[:, 1]
22
23print("Logistic weights:", clf.coef_[0])
24print("Logistic intercept:", clf.intercept_[0])
25print("Cross-entropy loss:", log_loss(y, prob))
26print("Probability of passing for [5.5, 6.0]:", clf.predict_proba([[5.5, 6.0]])[0, 1])

Strengths & Advantages

  • Outputs well-calibrated probabilities rather than just hard classes.
  • Very easy to interpret; feature weights show the log-odds impact of each variable.
  • Fast to train and serves as an excellent classification baseline.

Limitations & Drawbacks

  • Assumes a linear decision boundary in the feature space.
  • Can easily overfit if features are highly dimensional or collinear (requires regularization).
  • Cannot solve complex non-linear classification problems without manual feature engineering.

Real-World Case Studies

Predicting credit-card default from balance and income

Credit risk scoring
Scenario

A lender wants to estimate the probability that a customer will default on their credit-card debt, using features such as current balance, annual income, and student status (the Default dataset from An Introduction to Statistical Learning, with 10,000 customers). The positive class (default) is rare — only about 3% of customers — making it a classic imbalanced binary classification problem.

Approach

Fit a logistic regression Pr⁡(default=1)=σ(β0+β1⋅balance+β2⋅income+β3⋅student)\Pr(\text{default} = 1) = \sigma(\beta_0 + \beta_1 \cdot \text{balance} + \beta_2 \cdot \text{income} + \beta_3 \cdot \text{student}), inspect each coefficient as a log-odds effect, and convert the predicted probabilities to decisions via a tuned threshold.

Outcome

Balance is overwhelmingly the strongest predictor: its positive coefficient means the odds of default rise sharply with balance, and a customer with a 2,000balancehasadramaticallyhigherpredicteddefaultprobabilitythanonewitha2,000 balance has a dramatically higher predicted default probability than one with a 500 balance. Using the default 0.50.5 threshold the model reaches roughly 97% overall accuracy, but because only ~3% of customers default, that headline accuracy is close to the naive "predict no one defaults" baseline — so the lender instead lowers the threshold to catch more true defaulters, trading a higher false-positive rate for far fewer costly missed defaults. The case study underscores that for imbalanced classification, probability calibration and threshold choice matter more than raw accuracy.

Source: An Introduction to Statistical Learning (Ch. 4, Default data) — James, G., Witten, D., Hastie, T. and Tibshirani, R.

Common Misconceptions

MisconceptionLogistic regression is a regression algorithm that predicts continuous numbers.
CorrectionAlthough it uses linear regression math under the hood, it is a classification algorithm that predicts categorical class probabilities.
MisconceptionUsing mean squared error is fine for logistic regression training.
CorrectionMSE is non-convex when combined with the Sigmoid function, which makes optimization difficult due to local minima and flat gradients. Cross-entropy is convex and guarantees a single global minimum.

References & Further Reading

  1. The Elements of Statistical Learningtextbook

    By Hastie, T., Tibshirani, R. and Friedman, J

    View Resource →
  2. An Introduction to Statistical Learningtextbook

    By James, G., Witten, D., Hastie, T. and Tibshirani, R

    View Resource →
← Previous Topic
Linear Regression
Next Topic →
L1 & L2 Regularization