Decision Trees

Difficulty:Intermediate

Reading Time:25 min

Track:

Practitioner

A flowchart-like model that makes decisions by asking a series of yes/no questions about features.

ML PractitionerModule 9 of 17

Decision Trees

53%

TL;DR

A decision tree recursively partitions feature space with axis-aligned splits, asking one feature-threshold question per node until it reaches a leaf prediction.
Each split is chosen greedily to minimize child impurity, measured by Gini impurity $1 - \sum_i p_i^2$ or entropy $-\sum_i p_i \log_2 p_i$ .
Information gain is the drop in impurity from parent to the weighted average of the children; the algorithm picks the split with the largest gain.
Trees are highly interpretable and need no feature scaling, but a deep, unpruned tree overfits by memorizing noise and is high-variance.
Control overfitting with depth limits, minimum-samples constraints, or pruning; for raw accuracy, prefer ensembles (Random Forest, Gradient Boosting).

Learning Objectives

Explain how decision trees partition feature space recursively
Compute Gini Impurity and Entropy for a given set of labels
Calculate Information Gain to determine the optimal split feature and threshold
Describe the role of pruning and tree depth constraints in preventing overfitting

Intuition

How to think conceptually about this topic

Imagine playing the game "20 Questions" to guess an animal. You don't ask "Does it have a tail?" first, because that doesn't narrow it down much. You ask "Is it a mammal?" because that splits the animal kingdom in half. A Decision Tree uses math to figure out the absolute best sequence of questions to ask to arrive at the right answer as fast as possible.

Interactive Diagram

Test the intuition above by changing the model parameters

In Depth

Detailed explanations, contexts, and details

Decision Trees are non-parametric supervised learning models used for both classification and regression. They work by partitioning the feature space into distinct regions through a series of recursive binary splits.

Starting at the root node, the model checks a specific feature condition (e.g., "Is age < 30?"). If yes, it goes down the left branch; if no, it goes down the right branch. This process repeats at subsequent nodes until a leaf node is reached, which contains the final prediction.

Where is it used?

Decision Trees are widely used in business, finance, and medicine, where model interpretability and transparency are critical (e.g., explaining why a loan was denied, or determining a patient's risk category based on symptoms). They also serve as the fundamental building block for ensemble models like Random Forests and Gradient Boosting Machines.

How It Compares

Single Decision Tree vs Random Forest vs Gradient Boosting

Dimension	Single Decision Tree	Random Forest	Gradient Boosting
Bias / variance	Low bias, high variance	Low bias, variance reduced by averaging	Bias reduced sequentially, variance controlled by shrinkage
Interpretability	High — readable if/else rules	Moderate — needs feature-importance summaries	Low — many additive trees, needs SHAP/importances
Overfitting tendency	High if unpruned	Low — bagging averages out noise	Moderate — overfits if too many trees or high learning rate
Training cost	Cheapest — one tree	Moderate — many trees, but parallelizable	Higher — trees built sequentially
Typical accuracy	Baseline	Strong, robust out of the box	Often state of the art on tabular data when tuned

TakeawayUse a single tree when you need transparent rules; reach for Random Forest for a robust low-variance default, and Gradient Boosting when you want top tabular accuracy and can afford tuning.

When to Use It

Reach for this when

You need an interpretable model whose decisions can be read off as explicit if/else rules (e.g. explaining a loan denial).
The data has mixed feature types (numerical and categorical) and you want to skip feature scaling and most preprocessing.
You want to capture non-linear interactions automatically without manually engineering them.

Avoid it when

You need the highest possible accuracy on tabular data — a single tree is usually beaten by Random Forest or Gradient Boosting.
The true relationship is smooth or linear — axis-aligned splits approximate it as a clumsy staircase, so linear models do better.
The dataset is small or noisy and a deep tree would memorize it — at minimum, constrain depth or prune.

Rules of thumb

Always limit complexity: set a max depth or a minimum number of samples per leaf to fight overfitting.
Do not bother scaling features — splits are threshold-based and scale-invariant.
If accuracy matters more than interpretability, jump straight to an ensemble of trees.

Implementation

Reference code implementation

Python

model_fitting.py

1import numpy as np
2from sklearn.tree import DecisionTreeClassifier
3
4# Two-dimensional feature coordinates
5X = np.array([[0, 0], [1, 1.5], [1.5, 1], [8, 8], [8.5, 7.5], [9, 9]])
6# Binary Labels
7y = np.array([0, 0, 0, 1, 1, 1])
8
9# Initialize and fit Decision Tree
10dt = DecisionTreeClassifier(max_depth=2, random_state=42)
11dt.fit(X, y)
12print(f"Decision Tree Class for coordinate [7, 7]: {dt.predict([[7, 7]])[0]}")

Strengths & Advantages

Perfectly transparent and easy to interpret or visualize.
Requires little to no data preprocessing (no feature scaling needed).
Handles both numerical and categorical data naturally.

Limitations & Drawbacks

High tendency to overfit if not restricted (grows deep to memorize noise).
Unstable; small changes in the data can lead to a completely different tree structure.
Can struggle to capture complex linear relationships (creates staircase-like boundaries).

Real-World Case Studies

CART on the UCI German credit-risk benchmark

Credit risk / finance

Scenario

A lender wants to classify loan applicants as good or bad credit risks from 20 attributes (account status, loan duration, credit history, employment, etc.) on the 1,000-record UCI German Credit dataset. The model must be auditable: a regulator may ask why any individual applicant was declined.

Approach

Train a CART classification tree using Gini impurity to choose splits, then control overfitting by limiting tree depth and pruning low-information branches (cost-complexity pruning, as described in Breiman et al.). The fitted tree is read as a small set of human-legible rules, e.g. "if checking-account status is negative and loan duration exceeds 24 months, flag as high risk."

Outcome

A pruned single tree typically lands around 70-73% classification accuracy on held-out data on this benchmark — modestly below a Random Forest (roughly 76-78%) — but delivers explicit decision rules an analyst can audit and a regulator can review. The case illustrates the core practitioner trade-off: a single tree trades a few points of accuracy for transparency, and ensembles recover that accuracy at the cost of interpretability.

Source: Classification and Regression Trees — Breiman, L., Friedman, J., Stone, C.J. and Olshen, R.A.

Common Misconceptions

MisconceptionDecision Trees always require features to be scaled (e.g. Z-score normalization).

CorrectionUnlike distance-based models, Decision Trees split features individually at single thresholds, meaning scaling has absolutely no effect on split locations or tree performance.

MisconceptionA deeper tree is always more accurate.

CorrectionDeep trees can achieve 100% accuracy on training data but overfit significantly by memorizing noise. They perform poorly on unseen testing data unless regularized via depth limits or minimum sample splits.

References & Further Reading

Classification and Regression Treestextbook
By Breiman, L., Friedman, J., Stone, C.J. and Olshen, R.A
View Resource →
Machine Learning with Random Forests and Decision Treestextbook
By Scott, S
View Resource →

← Previous Topic

Support Vector Machines

Next Topic →

Ensemble Learning (RFs & GBMs)