Naive Bayes

Difficulty:Intermediate

Reading Time:30 min

Track:

Practitioner

A fast, probabilistic classifier that computes class probabilities using Bayes' Theorem with the 'naive' assumption of feature independence.

Prerequisites

Natural Language Processing

ML PractitionerModule 7 of 17

Naive Bayes

41%

TL;DR

Naive Bayes is a generative classifier: it models how each class generates features via $P(y)$ and $P(x_i \mid y)$ , then applies Bayes’ theorem.
The naive assumption is conditional independence of features given the class, giving $P(y \mid \mathbf{x}) \propto P(y)\prod_i P(x_i \mid y)$ .
Predict the class that maximizes the posterior; in practice work in log-space, $\arg\max_c \big[\log P(c) + \sum_i \log P(x_i \mid c)\big]$ , to avoid underflow.
Laplace (additive) smoothing adds $\alpha$ to every count so that an unseen feature value never forces a zero probability that wipes out the whole product.
It is a fast, data-efficient baseline that excels on high-dimensional sparse text (spam, sentiment) despite the independence assumption being false.
Probabilities are often poorly calibrated (over-confident) because correlated features double-count evidence, but the $\arg\max$ decision is frequently still correct.

Learning Objectives

Formulate the Naive Bayes classification model using Bayes' Theorem.
Explain the 'naive' conditional independence assumption and its mathematical implications.
Contrast Gaussian, Multinomial, and Bernoulli Naive Bayes variants and their use cases.
Apply Laplace smoothing to handle zero-frequency feature occurrences.
Explain the reason for computing probabilities in log-space to prevent underflow.

Intuition

How to think conceptually about this topic

Imagine you're trying to guess if an email is spam based on the words it contains: 'win', 'lottery', and 'money'. Calculating the exact probability of these words appearing together in a specific sequence is difficult. So, you make a simplifying, 'naive' assumption: you pretend that words appear completely independently of each other. You look up how likely 'win' is in spam, how likely 'lottery' is, and how likely 'money' is, and simply multiply those probabilities together. Even though this independence assumption is technically wrong, the resulting guess is usually correct!

Interactive Diagram

Test the intuition above by changing the model parameters

In Depth

Detailed explanations, contexts, and details

Naive Bayes classifiers are a collection of classification algorithms based on Bayes' Theorem. It is not a single algorithm but a family of algorithms where all of them share a common principle: every pair of features being classified is independent of each other. Despite their simplistic assumptions, Naive Bayes classifiers work extremely well in many complex real-world situations, particularly in document classification and spam filtering.

How It Compares

Naive Bayes vs Logistic Regression

Dimension	Naive Bayes	Logistic Regression
Modelling paradigm	Generative — models $P(\mathbf{x} \mid y)$ and $P(y)$ , then inverts with Bayes	Discriminative — models $P(y \mid \mathbf{x})$ directly
Feature independence	Assumes conditional independence given the class	No independence assumption; learns weights jointly
Small-data behavior	Data-efficient — converges fast, strong with few examples	Needs more data to reach its (lower) asymptotic error
Probability calibration	Often poorly calibrated / over-confident from correlated features	Typically well calibrated, especially with regularization
Training speed	Single pass of counting — extremely fast	Iterative optimization (gradient descent) — slower

TakeawayNaive Bayes wins when data is scarce and you need a fast baseline; logistic regression usually wins asymptotically with enough data and gives better-calibrated probabilities. A classic result (Ng & Jordan, 2001) shows NB reaches its error faster but logistic regression reaches a lower error eventually.

When to Use It

Reach for this when

Classifying text: spam filtering, sentiment, or topic labeling where the bag-of-words representation is high-dimensional and sparse.
You have little training data and need a model that estimates its few parameters reliably from a single pass.
You need a fast, cheap baseline to benchmark before investing in heavier models.
Features are roughly conditionally independent given the class, or correlations are mild enough that the $\arg\max$ decision survives.

Avoid it when

Features are strongly correlated (e.g. duplicate or near-duplicate columns) — NB double-counts evidence and polarizes probabilities.
You need well-calibrated probability estimates rather than just the most likely class — prefer logistic regression or a calibrated model.
The decision genuinely depends on feature interactions that independence erases — use trees or models that capture interactions.

Rules of thumb

Always apply Laplace smoothing ( $\alpha = 1$ is a sensible default) so unseen feature values cannot zero out a posterior.
Compute scores in log-space to avoid floating-point underflow on long documents.
Match the variant to the data: Multinomial NB for token counts, Bernoulli NB for binary presence/absence, Gaussian NB for continuous features.
Drop redundant or highly collinear features before fitting to reduce evidence double-counting.

Implementation

Reference code implementation

TypeScript

example.ts

1/**
2 * Simple Multinomial Naive Bayes Classifier for text tokens
3 */
4export class MultinomialNaiveBayes {
5  private classPriors: Record<string, number> = {};
6  private wordCounts: Record<string, Record<string, number>> = {};
7  private totalWordCounts: Record<string, number> = {};
8  private vocab: Set<string> = new Set();
9  private classes: string[] = [];
10
11  constructor(private alpha: number = 1.0) {}
12
13  fit(docs: string[][], labels: string[]) {
14    const numDocs = docs.length;
15    const classCounts: Record<string, number> = {};
16
17    // Count instances
18    for (let i = 0; i < numDocs; i++) {
19      const label = labels[i];
20      const doc = docs[i];
21
22      classCounts[label] = (classCounts[label] || 0) + 1;
23      if (!this.wordCounts[label]) {
24        this.wordCounts[label] = {};
25        this.totalWordCounts[label] = 0;
26      }
27
28      doc.forEach(word => {
29        this.vocab.add(word);
30        this.wordCounts[label][word] = (this.wordCounts[label][word] || 0) + 1;
31        this.totalWordCounts[label]++;
32      });
33    }
34
35    this.classes = Object.keys(classCounts);
36    this.classes.forEach(c => {
37      this.classPriors[c] = classCounts[c] / numDocs;
38    });
39  }
40
41  predict(doc: string[]): string {
42    let bestClass = this.classes[0];
43    let bestScore = -Infinity;
44    const vocabSize = this.vocab.size;
45
46    this.classes.forEach(c => {
47      // Start score with prior probability in log-space
48      let score = Math.log(this.classPriors[c]);
49      const totalWords = this.totalWordCounts[c];
50
51      doc.forEach(word => {
52        // Only count words that were seen in the training vocabulary
53        if (this.vocab.has(word)) {
54          const count = this.wordCounts[c][word] || 0;
55          // Apply Laplace smoothing
56          const prob = (count + this.alpha) / (totalWords + this.alpha * vocabSize);
57          score += Math.log(prob);
58        }
59      });
60
61      if (score > bestScore) {
62        bestScore = score;
63        bestClass = c;
64      }
65    });
66
67    return bestClass;
68  }
69}

Strengths & Advantages

Extremely fast to train and predict; requires only a single pass through the dataset.
Performs remarkably well on high-dimensional text classification (e.g. spam detection, sentiment analysis).
Requires relatively small amounts of training data to estimate parameters.

Limitations & Drawbacks

Known to be a poor estimator; the output class probabilities are often poorly calibrated.
The conditional independence assumption is fundamentally unrealistic for most natural datasets.

Real-World Case Studies

Bayesian spam filtering with a bag-of-words classifier

Email / NLP

Scenario

In the early 2000s, rule-based spam filters were brittle and easy for spammers to evade. Paul Graham’s essay "A Plan for Spam" proposed instead learning a per-user statistical filter that scores each incoming email from the words it contains, treating tokens as (naively) independent evidence for the Spam vs Ham classes.

Approach

Build a bag-of-words token model: estimate $P(\text{token} \mid \text{Spam})$ and $P(\text{token} \mid \text{Ham})$ from each user’s own corpus of good and spam mail, smooth rare tokens to avoid zero probabilities, and combine the most informative tokens via Bayes’ theorem to produce a spam probability for each new message.

Outcome

The naive-Bayes-style filter classified spam with very high accuracy — Graham reported catching about 99.5% of spam with roughly 0.03% false positives on his mail — dramatically better than the hand-written rules it replaced, and the approach (popularized as "Bayesian spam filtering") became the template for production spam filters. The lesson: a simple bag-of-words Naive Bayes model, trained per user and properly smoothed, is a remarkably strong text classifier.

Source: A Plan for Spam — Paul Graham

Common Misconceptions

MisconceptionNaive Bayes requires features to be truly independent in the real world to perform well.

CorrectionEven though features are rarely independent in reality (e.g. 'credit' and 'card' in spam), Naive Bayes often yields highly accurate classification decisions because the decision boundary (not the absolute probabilities) is what determines the final class assignment.

MisconceptionIf a feature is missing from a class in training data, the model will just treat its likelihood as very low.

CorrectionWithout smoothing, a zero-frequency count results in a likelihood of exactly 0. Because probabilities are multiplied together, a single zero likelihood will zero out the entire posterior probability, regardless of other evidence.

References & Further Reading

Speech and Language Processing (3rd ed. draft)textbook
By Dan Jurafsky and James H. Martin
View Resource →
Machine Learning: A Probabilistic Perspectivetextbook
By Kevin P. Murphy
View Resource →

Naive Bayes

Prerequisites

TL;DR

Learning Objectives

Intuition

Interactive Diagram

The Mathematics

In Depth

How It Compares

Naive Bayes vs Logistic Regression

When to Use It

Reach for this when

Avoid it when

Rules of thumb

Implementation

Strengths & Advantages

Limitations & Drawbacks

Real-World Case Studies

Bayesian spam filtering with a bag-of-words classifier

Common Misconceptions

Self-Check Quiz

References & Further Reading

Related Topics

Natural Language Processing

Logistic Regression