Sequence Models

Difficulty:Advanced

Reading Time:45 min

Track:

Deep Learning

Recurrent neural architectures designed to process sequential data, covering RNNs, LSTMs, GRUs, and the vanishing gradient problems associated with temporal dependencies.

Prerequisites

Neural Networks & Deep Learning

Deep LearningModule 8 of 19

Sequence Models

42%

TL;DR

Sequence models carry a hidden state across time steps, letting a network process variable-length inputs like text, audio, or time series.
Vanilla RNNs update $h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$ , but gradients computed via BPTT are a product of Jacobians, so they vanish or explode over long sequences.
LSTMs fix this with a protected cell state $C_t$ updated additively, $C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$ , giving gradients a near-unimpeded path backward through time.
GRUs merge the forget/input gates into a single update gate and drop the separate cell state, giving similar long-range performance to LSTMs with fewer parameters.
All recurrent models are inherently sequential at training and inference time, which is the main reason Transformers (fully parallel, attention-based) displaced them for most large-scale NLP tasks.

Learning Objectives

Differentiate sequence models from static feedforward architectures in terms of temporal state tracking.
Formulate and analyze the recurrence relations of vanilla Recurrent Neural Networks (RNNs).
Trace the flow of information through Long Short-Term Memory (LSTM) gates: forget, input, and output gates.
Explain the mechanics of Backpropagation Through Time (BPTT) and how vanishing gradients occur over long sequences.
Compare LSTMs and Gated Recurrent Units (GRUs) in terms of parameter efficiency and gate complexity.

Intuition

How to think conceptually about this topic

Imagine reading a sentence word by word. As you read, you don't forget the previous words; you carry a mental summary (a hidden state) in your head. For each new word, you update your understanding. Sequence models like RNNs mimic this behavior. A vanilla RNN simply combines the current input and the previous hidden state using a single layer. LSTMs improve on this by maintaining a separate, protected memory channel (the cell state). Inside an LSTM, gates act like valves: the forget gate decides what history to drop, the input gate decides what new info to write to the cell state, and the output gate decides what information to filter out as the final hidden state.

Interactive Diagram

Test the intuition above by changing the model parameters

In Depth

Detailed explanations, contexts, and details

Sequence models are neural network architectures optimized for sequential and temporal data. They process variable-length inputs by passing persistent hidden states across sequence steps, enabling applications in speech, translation, and time-series modeling.

How It Compares

Vanilla RNN vs LSTM vs GRU

Dimension	Vanilla RNN	LSTM	GRU
Gating mechanism	None — a single $\tanh$ layer combines $h_{t-1}$ and $x_t$ at every step.	Three gates (forget $f_t$ , input $i_t$ , output $o_t$ ) plus a candidate cell state $\tilde{C}_t$ .	Two gates (update $z_t$ , reset $r_t$ ); no separate cell state.
State maintained	Single hidden state $h_t$ .	Hidden state $h_t$ and a separate protected cell state $C_t$ .	Single hidden state $h_t$ (cell state folded in).
Parameter count (per layer, hidden size $n$, input size $m$)	$\approx n(n+m)$ — one weight matrix.	$\approx 4n(n+m)$ — four weight matrices.	$\approx 3n(n+m)$ — three weight matrices (~25% fewer than LSTM).
Long-range dependency capture	Poor — gradients vanish/explode after a few dozen steps.	Strong — additive cell-state path preserves gradient signal over hundreds of steps.	Strong — similar additive update gate gives comparable long-range performance to LSTM.
Training stability	Unstable on long sequences; needs careful initialization and gradient clipping.	Stable; the default choice historically for long-sequence tasks.	Stable; often converges slightly faster than LSTM due to fewer parameters.

TakeawayVanilla RNNs are simple but fail on anything but short sequences due to vanishing/exploding gradients. LSTMs fix this with a dedicated, additively-updated cell state at the cost of more parameters; GRUs recover most of that benefit with a leaner two-gate design, making them a good default when compute or data is limited.

When to Use It

Reach for this when

You are modeling sequential or temporal data (time series, audio, text, sensor streams) where order and recency genuinely matter.
Sequence lengths are short-to-moderate (roughly tens to low hundreds of steps) so vanishing/exploding gradients are manageable, especially with LSTMs/GRUs.
You need an online/streaming model that can update its state incrementally one step at a time without re-processing the whole sequence.
Compute or memory budgets are tight and full self-attention (quadratic in sequence length) is infeasible.

Avoid it when

Sequences are very long (thousands of steps) and you need to model long-range dependencies efficiently — Transformers with attention typically outperform recurrent models here.
You need parallel training across time steps for speed; the inherently sequential recurrence of RNN/LSTM/GRU prevents this.
You have abundant compute and large labeled datasets, where attention-based architectures usually achieve better accuracy with similar or less wall-clock training time.
Interpretability of "what the model attends to" is important — attention weights are generally easier to inspect than recurrent hidden-state dynamics.

Rules of thumb

Always apply gradient clipping (e.g. clip norm to 1.0–5.0) when training vanilla RNNs or LSTMs/GRUs on long sequences.
Prefer LSTM when you suspect very long dependencies matter and you have enough data/compute; prefer GRU as a lighter, often-just-as-good default.
If sequences exceed a few hundred steps, consider truncated BPTT, hierarchical RNNs, or switching to an attention-based architecture entirely.

Implementation

Reference code implementation

TypeScript

example.ts

1/**
2 * Simple Recurrent Neural Network Cell Implementation
3 */
4export class RNNCell {
5  constructor(
6    public inputDim: number,
7    public hiddenDim: number,
8    public Whh: number[][], // hiddenDim x hiddenDim
9    public Wxh: number[][], // hiddenDim x inputDim
10    public bh: number[]     // hiddenDim
11  ) {}
12
13  /**
14   * Performs forward step computing next hidden state
15   */
16  step(x: number[], prevH: number[]): number[] {
17    const nextH = new Array(this.hiddenDim).fill(0);
18    
19    for (let i = 0; i < this.hiddenDim; i++) {
20      let sum = this.bh[i];
21      
22      // Hidden contribution
23      for (let j = 0; j < this.hiddenDim; j++) {
24        sum += this.Whh[i][j] * prevH[j];
25      }
26      
27      // Input contribution
28      for (let j = 0; j < this.inputDim; j++) {
29        sum += this.Wxh[i][j] * x[j];
30      }
31      
32      // Tanh activation
33      nextH[i] = Math.tanh(sum);
34    }
35    
36    return nextH;
37  }
38
39  /**
40   * Unrolls RNN cell over a sequence of inputs
41   */
42  unroll(inputs: number[][], initH: number[]): number[][] {
43    const states: number[][] = [];
44    let currentH = [...initH];
45    
46    for (const x of inputs) {
47      currentH = this.step(x, currentH);
48      states.push(currentH);
49    }
50    
51    return states;
52  }
53}

Strengths & Advantages

Capable of processing variable-length inputs without requiring fixed input dimensions.
Maintains structural temporal order and captures sequence context dynamically.
LSTMs can successfully capture dependencies across long intervals of time.

Limitations & Drawbacks

Sequential training is slow and cannot be easily parallelized like feedforward layers.
Prone to vanishing and exploding gradients when processing long contexts.
Prone to catastrophic forgetting, where newer inputs overwrite older historical context.

Real-World Case Studies

The original LSTM paper: solving tasks vanilla RNNs could not

Sequence learning research

Scenario

In 1997, Hochreiter and Schmidhuber observed that standard RNNs trained with BPTT or real-time recurrent learning failed to learn synthetic benchmark tasks requiring memory over very long time lags — for example, tasks needing a signal to be remembered for 1,000 or more time steps before it became relevant again, since the relevant error signal had vanished long before reaching that far back.

Approach

They introduced the Long Short-Term Memory architecture, with a protected, linear cell-state update path gated by learned multiplicative units (the precursors to the modern forget/input/output gates), explicitly designed so that the backward gradient through the cell state would neither vanish nor explode by default — addressing the analysis that constant error flow requires the recurrent weight on the memory path to be approximately 1.

Outcome

On their benchmark long-time-lag tasks, LSTM solved problems involving minimum time lags of 1,000 steps or more, where contemporary RNN variants of the time either failed to converge or required orders of magnitude more training updates. This result established LSTMs as the dominant recurrent architecture for roughly the next two decades, underpinning major advances such as Google’s neural machine translation systems before the shift to Transformers around 2017.

Source: Long Short-Term Memory — Hochreiter, S. and Schmidhuber, J.

Common Misconceptions

MisconceptionVanilla Recurrent Neural Networks can easily learn very long-term dependencies in sequences.

CorrectionVanilla RNNs struggle to learn long-term dependencies (beyond a few dozen steps) because gradients fade exponentially as they are multiplied repeatedly backward through time, a problem that motivated the creation of LSTMs and GRUs.

MisconceptionLSTMs completely eliminate the vanishing gradient problem.

CorrectionWhile LSTMs greatly alleviate vanishing gradients by establishing a linear cell state flow with additive updates, they do not completely eliminate the issue for extremely long sequences or when parameters are poorly initialized.

References & Further Reading

Understanding LSTM Networkstutorial
By Christopher Olah
View Resource →
Supervised Sequence Labeling with Recurrent Neural Networkstextbook
By Alex Graves
View Resource →

Sequence Models

Prerequisites

TL;DR

Learning Objectives

Intuition

Interactive Diagram

The Mathematics

In Depth

How It Compares

Vanilla RNN vs LSTM vs GRU

When to Use It

Reach for this when

Avoid it when

Rules of thumb

Implementation

Strengths & Advantages

Limitations & Drawbacks

Real-World Case Studies

The original LSTM paper: solving tasks vanilla RNNs could not

Common Misconceptions

Self-Check Quiz

References & Further Reading

Related Topics

Neural Networks & Deep Learning

Natural Language Processing

Transformers