Embeddings & Tokenization

Embeddings and Tokenization

Difficulty:Intermediate

Reading Time:35 min

Track:

Deep Learning

Techniques for dividing raw text into tokens and mapping them to dense, continuous vector spaces that capture semantic similarities.

Prerequisites

Natural Language Processing

Deep LearningModule 7 of 19

Embeddings and Tokenization

37%

TL;DR

Tokenization splits raw text into discrete units; modern LLMs use subword tokenization (BPE, WordPiece) to balance vocabulary size against out-of-vocabulary coverage.
An embedding maps each token ID to a dense, learned vector $\mathbf{e} \in \mathbb{R}^d$ that places semantically related tokens near each other in space.
Embedding lookup is just selecting a row of the matrix $\mathbf{E}$ — equivalently a one-hot vector times $\mathbf{E}$ .
Semantic similarity is measured with cosine similarity $\cos(\theta) = \frac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{u}\|\|\mathbf{v}\|}$ , which compares direction and ignores magnitude.
Static embeddings (word2vec, GloVe) give one vector per word; contextual embeddings (BERT) give a different vector per occurrence, resolving polysemy.

Learning Objectives

Contrast character, word, and subword tokenization in terms of vocabulary size and out-of-vocabulary handling.
Explain the mechanics of Byte Pair Encoding (BPE) for building token vocabularies.
Formulate embedding lookup as a matrix multiplication of a one-hot vector and a weight matrix.
Calculate semantic similarity between embeddings using cosine similarity.
Analyze semantic bias and representation drift in high-dimensional embedding spaces.

Intuition

How to think conceptually about this topic

Computers do not understand text; they only understand numbers. To solve this, we must first break text into pieces called tokens (which can be whole words, syllables, or letters). This is tokenization. Next, we assign each token a vector (a list of coordinates in a high-dimensional space). This is an embedding. If two tokens have similar meanings (like 'king' and 'queen'), their vectors will point in similar directions. This allows the model to perform mathematical operations on semantics, turning language understanding into geometry.

Interactive Diagram

Test the intuition above by changing the model parameters

In Depth

Detailed explanations, contexts, and details

Embeddings and Tokenization form the interface between raw textual characters and deep learning architectures. Tokenization decomposes character sequences into integer IDs, while embeddings map those integer IDs into continuous vector manifolds that capture semantic context.

How It Compares

One-Hot vs Static vs Contextual Representations

Dimension	One-Hot Encoding	Word2Vec/Static Embeddings	Contextual Embeddings (BERT)
Dimensionality	$V$ (vocabulary size, e.g. 50k) — sparse	Low, dense (e.g. 100–300)	Low, dense (e.g. 768–1024)
Captures meaning?	No — all tokens equidistant, no semantics	Yes — similar words get nearby vectors	Yes — rich semantics plus syntax
Context-sensitivity	None	None — one fixed vector per word (polysemy unresolved)	Yes — vector changes with the surrounding sentence
Typical use case	Tiny vocabularies, classical ML baselines	Lightweight semantic search, pre-deep-learning NLP	Modern NLP: QA, NER, retrieval, fine-tuning

TakeawayMove from one-hot (no meaning) to static embeddings (meaning, but one vector per word) to contextual embeddings (meaning that adapts to context) as your need to resolve polysemy and capture nuance grows — at the cost of more compute.

When to Use It

Reach for this when

You need to feed text into a neural network and want dense inputs that encode semantic similarity rather than sparse one-hot IDs.
You are building semantic search, retrieval, clustering, or deduplication where cosine similarity between vectors ranks relevance.
You face out-of-vocabulary or morphologically rich text and want subword tokenization (BPE/WordPiece) to avoid [UNK] tokens.

Avoid it when

The vocabulary is tiny and fixed and the categories carry no inherent similarity — plain one-hot or a learned lookup is simpler.
You need to disambiguate polysemous words by context but only have static (word2vec/GloVe) embeddings — reach for contextual models like BERT instead.
Your corpus is too small to learn meaningful geometry — embeddings trained from scratch on little data will be noisy; use pretrained vectors or transfer learning.

Rules of thumb

Normalize embeddings to unit length so cosine similarity reduces to a plain dot product and is faster to compute at scale.
Pick a vocabulary of roughly 32k–100k subword tokens for general-purpose language models to balance sequence length and parameters.
Embedding dimension scales with task complexity and data size: a few hundred for static vectors, 768+ for large contextual models.

Implementation

Reference code implementation

TypeScript

example.ts

1/**
2 * Simple Character-level Byte Pair Encoding (BPE) Tokenizer
3 */
4export class BPETokenizer {
5  private vocab: Map<string, number> = new Map();
6  private merges: Map<string, string> = new Map();
7
8  constructor() {}
9
10  /**
11   * Fits vocabulary by merging most frequent adjacent character pairs
12   */
13  fit(corpus: string, maxVocabSize: number) {
14    // Start with individual characters as vocabulary
15    const chars = Array.from(new Set(corpus));
16    chars.forEach((c, idx) => this.vocab.set(c, idx));
17    
18    let currentVocabSize = chars.length;
19    let splits = corpus.split("").filter(c => c.trim().length > 0);
20
21    while (currentVocabSize < maxVocabSize) {
22      const pairCounts = new Map<string, number>();
23      
24      for (let i = 0; i < splits.length - 1; i++) {
25        const pair = splits[i] + "," + splits[i + 1];
26        pairCounts.set(pair, (pairCounts.get(pair) || 0) + 1);
27      }
28
29      if (pairCounts.size === 0) break;
30
31      // Find most frequent pair
32      let bestPair = "";
33      let maxCount = 0;
34      for (const [pair, count] of pairCounts.entries()) {
35        if (count > maxCount) {
36          maxCount = count;
37          bestPair = pair;
38        }
39      }
40
41      if (maxCount < 2) break;
42
43      const [p1, p2] = bestPair.split(",");
44      const merged = p1 + p2;
45      
46      this.merges.set(p1 + "," + p2, merged);
47      this.vocab.set(merged, currentVocabSize++);
48
49      // Perform merges in our splits array
50      const newSplits: string[] = [];
51      let i = 0;
52      while (i < splits.length) {
53        if (i < splits.length - 1 && splits[i] === p1 && splits[i + 1] === p2) {
54          newSplits.push(merged);
55          i += 2;
56        } else {
57          newSplits.push(splits[i]);
58          i++;
59        }
60      }
61      splits = newSplits;
62    }
63  }
64
65  tokenize(text: string): number[] {
66    const tokens: number[] = [];
67    const textChars = text.split("");
68    
69    // Simple greedy lookup of substrings in vocabulary
70    let i = 0;
71    while (i < textChars.length) {
72      let longestMatch = "";
73      let matchIdx = -1;
74      
75      for (let len = 1; len <= textChars.length - i; len++) {
76        const substr = textChars.slice(i, i + len).join("");
77        if (this.vocab.has(substr)) {
78          longestMatch = substr;
79          matchIdx = this.vocab.get(substr) ?? -1;
80        }
81      }
82
83      if (matchIdx !== -1) {
84        tokens.push(matchIdx);
85        i += Math.max(1, longestMatch.length);
86      } else {
87        // Fallback for character not in vocabulary
88        tokens.push(-1);
89        i++;
90      }
91    }
92    return tokens;
93  }
94}

Strengths & Advantages

Subword tokenizers completely eliminate out-of-vocabulary issues.
Embeddings compress sparse, high-dimensional words into dense, dense vector structures.
Cosine similarity enables fast semantic search and document retrieval.

Limitations & Drawbacks

Subword tokenizers can split words in non-intuitive ways that complicate interpretation.
Embeddings require massive datasets to learn accurate semantic relationships.
Static embeddings fail to handle polysemy (words with multiple meanings depending on context).

Real-World Case Studies

Word2Vec: learning analogies from raw text at scale

Natural language processing

Scenario

Before 2013, word representations were largely sparse one-hot or count-based vectors that captured no semantic similarity and scaled poorly. Mikolov and colleagues at Google wanted dense word vectors learned efficiently from very large unlabeled corpora (on the order of billions of words).

Approach

They introduced the Skip-gram and Continuous Bag-of-Words (CBOW) architectures — shallow neural networks trained to predict a word from its context (or vice versa) — which strip out the hidden non-linearity to train on huge datasets cheaply. The learned vectors were evaluated on a word-analogy benchmark using vector arithmetic such as $\text{king} - \text{man} + \text{woman}$ and nearest-neighbour cosine similarity.

Outcome

Training on a 1.6-billion-word corpus, the models learned 300-dimensional vectors in which linear analogies emerged: the system answered semantic/syntactic analogy questions at roughly 60% accuracy (versus near-zero for earlier baselines), while training orders of magnitude faster than prior neural language models. This established that meaning can be captured as geometry in a dense vector space and seeded the modern embedding era.

Source: Efficient Estimation of Word Representations in Vector Space — Mikolov, T., Chen, K., Corrado, G. and Dean, J.

Common Misconceptions

MisconceptionTokenization is a simple rule-based split on whitespace and punctuation marks.

CorrectionWhile simple tokenizers split on spaces, modern language models use subword tokenization algorithms like Byte Pair Encoding (BPE) or WordPiece to handle compound words and out-of-vocabulary terms by splitting them into common subword prefixes or suffixes.

MisconceptionEmbeddings capture a single, objective, and unbiased definition of words.

CorrectionEmbeddings reflect whatever associations exist in their training data. If the corpus contains societal or historical biases, the resulting spatial relationships (e.g. vector math like King - Man + Woman = Queen) will reflect and encode those biases.

References & Further Reading

Speech and Language Processing (Chapter 2 & 6)textbook
By Dan Jurafsky and James H. Martin
View Resource →
Efficient Estimation of Word Representations in Vector Spacepaper
By Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean
View Resource →

Embeddings and Tokenization

Prerequisites

TL;DR

Learning Objectives

Intuition

Interactive Diagram

The Mathematics

In Depth

How It Compares

One-Hot vs Static vs Contextual Representations

When to Use It

Reach for this when

Avoid it when

Rules of thumb

Implementation

Strengths & Advantages

Limitations & Drawbacks

Real-World Case Studies

Word2Vec: learning analogies from raw text at scale

Common Misconceptions

Self-Check Quiz

References & Further Reading

Related Topics

Natural Language Processing

Transformers