Deep Learning Handbook

Theme

RNN and Sequence Modeling

Quick overview Recurrent Neural Networks are recursive architectures designed for sequential data—the foundation for language models, machine translation, and speech recognition. This article breaks down RNN unrolling, BPTT and vanishing gradients, the LSTM three-gate structure and GRU, compares them to Transformers, and reviews remaining use cases like streaming speech and online prediction.

RNN and Sequence Modeling ​

In one sentence: A Recurrent Neural Network (RNN) is a network that shares weights across time, passing past inputs forward step by step through a "hidden state" to explicitly model sequential dependencies—it was the first major architecture for processing "ordered data" in deep learning (for a big-picture view, read Anatomy of Deep Learning Architectures first).

1. Why Sequences Need Specialized Networks ​

Language, speech, video, and time series share one thing: elements have an order, and that order carries meaning. "I hit you" and "you hit me" use the exact same three characters but have completely different meanings; today's and yesterday's stock prices are correlated. There are two naive approaches to handling sequences with ordinary neural networks:

  1. Concatenate the entire sequence into one long vector—variable lengths don't align, and positional structure is lost.
  2. Use a sliding window of fixed length—long-range dependencies outside the window are invisible, and within the window, it's just "feature concatenation" rather than "temporal reasoning."

RNNs answer with recurrence: the same weights (W) are reused at every time step, combining the previous hidden state $h_{t-1}$ with the current input $x_t$ to compute a new state $h_t$. This gives the network "memory," and the parameter count stays constant regardless of sequence length—this is the weight-sharing idea applied to the time dimension (the spatial version is covered in CNN and Computer Vision).

2. RNN Mechanics and Unrolling ​

Computation at a single time step (tanh version as example):

$$ h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b) $$

Output at each step (if needed): $y_t = W_{hy} h_t + b_y$. Unfolding an RNN across time produces a feedforward network where "the same parameters are replicated at every moment":

x1 → h1 → x2 → h2 → x3 → h3 → … → xt → ht → yt

After unrolling, an RNN becomes "a very deep network with shared weights," where the depth equals the sequence length. This perspective is crucial—it explains the mathematical essence of RNNs and foreshadows their fate (see the BPTT section below).

RNN variants fall into four categories based on their input-output relationships: many-to-one (sentence-level sentiment classification), one-to-many (image captioning), many-to-many synchronous (frame-by-frame annotation), and many-to-many asynchronous (encoder-decoder for machine translation).

3. BPTT and Vanishing Gradients ​

RNNs are trained using Backpropagation Through Time (BPTT): first unroll the sequence into a feedforward graph, then apply the chain rule from Backpropagation and Automatic Differentiation to compute gradients, summing the contributions of all time steps for the shared weight W.

The problem lies in the chained multiplication from the chain rule. For an error signal to propagate backward through $k$ time steps, the gradient contains $k$ factors of $\tanh'$ (≤ 1) and repeated multiplication by $W$:

$$ \frac{\partial L}{\partial h_{t-k}} \propto \prod_{i=1}^{k} \text{diag}(\tanh'(h_{t-i})) , W_{hh} $$

When the largest singular value of $W_{hh}$ is < 1, the product decays exponentially with $k$ → vanishing gradient, meaning the network can't remember distant information; > 1 leads to exponential explosion → exploding gradient (though gradient clipping can mitigate this). As a result, a classic RNN's effective memory is only about 5–10 steps. This explains why RNNs struggle to learn long-range dependencies like "keeping subject-verb agreement across dozens of words."

The Root Cause

Vanishing gradient isn't a "bug"—it's a mathematical inevitability of deep, weight-shared architectures: any architecture that propagates error signals through many layers of nonlinear multiplication will encounter this. RNNs are depth = sequence length, so they hit it first. Much of the mitigation is covered in Initialization and Normalization, but structural fixes require LSTM.

4. LSTM Three-Gate Structure and GRU ​

LSTM (Long Short-Term Memory, 1997) ​

LSTM's core idea is adding a "cell state conveyor belt" $C_t$: information flows straight along the time axis, allowing error signals to propagate over long distances without degradation. Its control mechanism is three gates (all using sigmoid to output 0–1 "openness" values, and tanh to generate candidate values):

  • Forget gate $f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)$: decides how much of the old cell state to keep.
  • Input gate $i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)$: decides how much new information to write.
  • Output gate $o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)$: decides how much of the current state to expose externally.

Cell state update: $C_t = f_t \odot C_{t-1} + i_t \odot \tilde C_t$. Hidden state: $h_t = o_t \odot \tanh(C_t)$. Focus on the cell state line: it involves only element-wise multiply-and-add, allowing error to flow back almost unchanged, extending the effective memory window from 5–10 steps to hundreds.

GRU (Gated Recurrent Unit, 2014) ​

GRU is a "simplified" version of LSTM: it merges the three gates into two—update gate and reset gate—and doesn't maintain a separate cell state, reducing parameters by about 1/3 compared to LSTM. In practice, GRU and LSTM have comparable performance with faster training, and GRU is often preferred on small datasets. Rule of thumb: choose GRU for small data, choose LSTM when you need extreme long-range modeling or already have LSTM infrastructure. In nearly all cases, however, both will be outperformed by larger-scale Transformer architectures.

5. Sequence Tasks ​

  • Language modeling: Trained on "predict the next word," it's the neural version of N-grams and the prototype for later Large Language Models (LLMs). RNN-LMs remember context via hidden states, paired with softmax to output vocabulary probabilities (output layer design in Loss Functions and Output Layers).
  • Machine translation: The encoder "reads" the source language into a semantic vector; the decoder generates the target language word by word. Bahdanau attention (2015) let the decoder "look back" at key positions in the encoder at each step—a mechanism later pushed to its limit by the Transformer (see Attention Mechanisms).
  • Speech recognition/synthesis: RNNs/LSTMs were the backbone of speech models for years (WaveNet also uses dilated causal convolutions). They still have applications in streaming speech scenarios today—see Speech and Audio.
  • Time series prediction: Sales, traffic, stock forecasting—RNNs remain one of the common baselines alongside CNNs and Transformers.

6. Comparison with Transformers ​

The Transformer (2017) replaced RNNs on virtually all sequence tasks. Understanding "why" matters:

DimensionRNN / LSTMTransformer
ParallelismSequential: $h_t$ depends on $h_{t-1}$, cannot parallelizeSelf-attention sees the whole sentence at once, fully parallelizable
Long-range dependenciesMitigated by gating, but still limited by path lengthAny two positions have path length 1, modeled directly
Computational complexity (seq. length n)O(1) per step, O(n) total, supports streamingO(n²) per layer, expensive for long sequences (with FlashAttention and other optimizations)
Inductive biasNatural sense of "order," friendlier to small dataRelies on positional encoding to inject order
CostCheap, deployable on edge devicesLarge models require massive compute

RNNs lost on "sequential execution": GPUs can't leverage their parallelism, and training becomes extremely slow for long sequences, while attention's O(n²) is perfectly acceptable for moderate lengths. But RNNs have properties that Transformers lack: fixed-size state, constant inference cost, and native streaming capability—this is why they survive today (see next section).

7. Remaining Use Cases ​

"Transformers replaced RNNs" is the dominant narrative, but the engineering world is pluralistic:

  • Streaming speech recognition: Real-time recognition requires "current output to depend only on already-seen input." Causal structures like RNNs/TDNNs are naturally suited, enabling real-time inference with constant latency.
  • Online prediction and edge deployment: On-device time series forecasting (heart rate, sensors), incremental online learning—RNNs have small state, no quadratic complexity, and are more resource-efficient than attention.
  • Theoretical counterpoint for state space models: New models like Mamba revive the RNN idea of "fixed-length state, linear complexity," serving as a modern RNN renaissance—a reminder that architectures go out of fashion, but ideas cycle back.

Learning Advice

RNNs/LSTMs may no longer be the "main weapon" on your resume, but they distill the three core things in deep learning: weight sharing, time unrolling, and the essence of vanishing gradients. Master these, and understanding Transformer Pre-Norm, residual connections, and KV cache (see Transformer Architecture) will be much smoother.

8. Trade-offs ​

  • RNN vs. attention: Long sequences requiring streaming → RNN/state space models; offline training with strong long-range modeling → attention.
  • LSTM vs. GRU: Parameters and performance are nearly equivalent. Pick whichever has a simpler implementation and better ecosystem support—don't overthink it.
  • Unidirectional vs. bidirectional: Bidirectional LSTMs see future context and perform better, but can't support streaming—online systems for speech/translation often have to fall back to unidirectional.
  • Depth and training difficulty: Stacking multiple LSTM layers has diminishing returns and still faces gradient issues. Always have gradient clipping and appropriate initialization ready (see Training Recipes and Hyperparameter Tuning).

Further Reading ​

References ​