Transformer Architecture Deep Dive

Transformer is a sequence model architecture built entirely on attention mechanisms: it lets any two positions in a sequence directly "see" each other, modeling global dependencies in parallel. When Google's team proposed it in 2017 in Attention Is All You Need, the target was machine translation; today, it's the backbone of every mainstream LLM — GPT, Llama, Qwen, DeepSeek. Understanding every component of Transformer is the prerequisite for understanding all LLMs.

One-line summary: Transformer = self-attention (modeling relationships between positions) + feedforward network (per-position non-linear transformation) + residual connections and normalization (stable training), stacked in alternating deep networks. Its combination with language modeling forms the engine of "big models."

1. Why Attention: From RNN to Transformer

RNN/LSTM must compress history into a single hidden state vector, compute sequentially step-by-step, can't parallelize, and long-distance information gets easily diluted. Self-attention lets every position directly compute correlation weights with all other positions, solving both problems simultaneously:

Dimension	RNN/LSTM	Transformer
Parallelism	Step-by-step serial	Full sequence parallel
Long-range dependencies	Compressed into hidden state, easily lost	Direct connections between any two positions
Complexity (per layer)	O(n·d²) serial	O(n²·d) parallel
Sequence length scalability	Theoretically infinite, practically limited	Limited by O(n²) attention constraint (see Context & Long Context)

2. Self-Attention and QKV: Step-by-Step Breakdown

1. Where Q, K, V Come From

For each token vector $$x_i$$ in the input sequence, multiply by three weight matrices to get Query, Key, Value three vectors:

For each input token x ∈ R^{d_model}:
  Q = x · W_q    (Query: what am I looking for?)
  K = x · W_k    (Key: what am I / what clues do I provide?)
  V = x · W_v    (Value: the actual content to aggregate)

W_q, W_k ∈ R^{d_model × d_k}, W_v ∈ R^{d_model × d_v}
(d_k = d_v = d_model / num_heads, e.g., 4096/32 = 128)

Intuition: Q represents "what this position wants to find from elsewhere", K represents "what this position shows outward as a label", and V represents "this position's actual content". Attention weights = similarity computed using Q vs. all Ks; output = weighted sum of Vs.

2. Scaled Dot-Product Attention

Attention(Q, K, V) = softmax( Q·K^T / √d_k ) · V

Matrix shapes: Q[n, d_k], K^T[d_k, n] → score matrix [n, n]
Row i, column j = dot product of q_i and k_j (similarity),
after softmax it becomes "position i's attention weight for position j",
finally multiplied by V to get position i's weighted output [n, d_v]

Why divide by √d_k? When q and k components are approximately independent zero-mean with variance 1, the dot product's variance is proportional to d_k. Higher dimensions → larger dot product values → softmax enters saturation zone faster (gradients approach zero). Dividing by √d_k normalizes variance, keeping attention scores in a "well-behaved" distribution for more stable training.

Walk through a complete 2-token example (for intuitive feel): set two token vectors as x₁, x₂, d_k = 2. Compute Q = [q₁, q₂], K = [k₁, k₂], first take dot products to get score matrix [q₁·k₁, q₁·k₂; q₂·k₁, q₂·k₂] (2×2), divide by √2 for scaling, then softmax each row — row i's two weights sum to 1, representing "how much information position i takes from position 1 and position 2". Finally, apply these two rows of weights to V's rows and sum, producing two output vectors. The whole process has no loops, expressed entirely by matrix multiplication — which is precisely why it can be highly parallelized on GPUs.

# Single-head self-attention implementation (PyTorch style, illustrative)
import torch, torch.nn.functional as F

def scaled_dot_product_attention(q, k, v, mask=None):
    # q: [n, d_k], k: [n, d_k], v: [n, d_v], mask: [n, n] (optional)
    d_k = q.size(-1)
    scores = q @ k.transpose(-2, -1) / (d_k ** 0.5)   # [n, n] dot product + scaling
    if mask is not None:
        scores = scores.masked_fill(mask == 0, float("-inf"))  # causal mask
    attn = F.softmax(scores, dim=-1)                   # normalize per row
    out = attn @ v                                     # [n, d_v]
    return out, attn

A one-image memory aid

Self-attention is three steps: Q asks, K answers (compute weights), V weighted sum (take content). All attention variants (multi-head, causal, FlashAttention) are engineering modifications on these three steps.

3. Multi-Head Attention: Let "Relationships" Be Seen Separately

Split the d_model space into h heads, each doing attention in an independent low-dimensional subspace (d_k = d_model/h), then concatenate and project:

MultiHead(Q, K, V) = Concat(head_1, ..., head_h) · W_o
head_i = Attention(x·W_q^i, x·W_k^i, x·W_v^i)

h = 8 or 32 (e.g., GPT-2 has 12 heads, GPT-3 has 96 heads, Llama 2 70B has 64 heads)

Why multi-head? A single-head attention can only learn "one type of similarity relationship," but language relationships are diverse: some heads handle syntactic proximity, some handle coreference, some handle positional information. Multi-head = multiple "perspectives" observing the same sequence in parallel, each handling one type of relationship, then fused by W_o. Additionally, low-dimensional subspaces make per-head computation lighter and matrices more parallel, training more efficiently. This is also the key to attention's parallelism.

Typical hyperparameters across generations of models illustrate how "depth, width, head count" co-vary (values subject to official releases):

Model	Parameters	Layers	Hidden dim d_model	Head count	d_k
GPT-2 XL (2019)	1.5B	48	1600	25	64
GPT-3 (2020)	175B	96	12288	96	128
Llama 2 7B (2023)	7B	32	4096	32	128
Llama 2 70B (2023)	70B	80	8192	64	128
Qwen2.5 72B (2024)	72B	80	8192	64	128

Note how d_k is highly consistent (often 64 or 128) — the scaling factor √d_k and dimension design empirical values are quite stable across the industry.

4. Positional Encoding: Letting the Model Know "Who's Before, Who's After"

Self-attention is permutation-equivariant — swap any two tokens, and the output swaps accordingly, completely losing order information. Position must be explicitly injected. Three major approaches:

Approach	Mechanism	Extrapolation	Representative
Absolute positional encoding	Add a learnable/sine vector per position	Poor (no encoding outside training window)	Original Transformer, BERT
Relative positional encoding	Add "position offset j−i" bias in scores	Good	T5, Transformer-XL
Rotary Position Embedding (RoPE)	Rotate Q/K vectors by position-dependent angle, scores contain relative j−i info	Moderate (good within training window, degrades outside)	Llama, Qwen, DeepSeek
Linear bias (ALiBi)	Directly subtract a distance-linearly-increasing penalty from scores	Good (strong length extrapolation)	MPT, early long-context models

The one-line difference between three approaches: absolute encoding "writes position into the input vector" (addition), requiring the model to learn to use it; relative encoding "writes position into the attention score" (explicit bias), more aligned with the intuition that "attention cares about relative distance"; RoPE "writes position into Q/K rotation" (multiplication), preserving relative semantics without adding extra memory or parameters.

1. RoPE: The Modern Mainstream

Rotary Position Embedding (RoPE) by Su et al. 2021: rotate every 2D block of q, k vectors by position angle, making attention scores depend only on the relative distance j−i between two positions, and naturally carrying an inductive bias that "more distant positions get lower weights":

For each 2D block (x1, x2) of q/k at position m:
  Rotation angle θ_m = m · base^{-2i/d_k} (i is dimension index, base often 10000)
  (x1', x2') = (x1·cos θ_m − x2·sin θ_m, x1·sin θ_m + x2·cos θ_m)
Effect: <q_m, k_n> = <R_m q, R_n k> = f(m − n), i.e., relative position

RoPE became mainstream because: cheap to implement (only touches Q/K), good relative-position semantics, and extensible to longer contexts via interpolation (see Context & Long Context).

Intuitively, RoPE can be understood as "giving each position's Q/K a knob": the knob's angle grows linearly with position, different dimensions rotate at different frequencies (low-frequency dims handle long-range info, high-frequency dims handle local details). When two positions' vectors are dot-producted, the rotation angle difference exactly equals the position difference — thus "relative distance" is cleanly encoded into attention scores mathematically, without any extra positional parameter table.

2. Positional Encoding Extrapolation Problems

During training, the model only sees positions within the window (e.g., 4096); beyond that, RoPE's angle range expands dramatically and the attention score distribution breaks — this is the structural root of "the model starts babbling past 4096 tokens." ALiBi natively extrapolates well because its penalty is linear; RoPE needs interpolation/expansion tricks. This is the central topic of long-context research.

5. Residual Connections and LayerNorm: Pre-Norm vs Post-Norm

Each sub-layer (attention, FFN) wraps in "residual + normalization" structure, ensuring deep network gradients flow and numerical stability:

Post-Norm (original Transformer):
  x' = LayerNorm( x + Sublayer(x) )
Pre-Norm (mainstream since GPT-2):
  x' = x + Sublayer( LayerNorm(x) )

Comparison	Post-Norm	Pre-Norm
Normalization position	After residual addition	Before sub-layer entry
Training stability	Poor (deep networks easily diverge)	Good (residual path is clean, gradients flow easily)
Mainstream status	Historical approach	De facto standard for modern LLMs (GPT/Llama/Qwen)

Why Pre-Norm won

Pre-Norm lets each residual branch pass directly to the output, equivalent to a "cleaner identity path," and gradients don't back-propagate through the normalization layer, so the network can be deepened without lowering learning rate. The trade-off: certain implementations yield slightly inferior deep-layer performance compared to tuned Post-Norm, but the stability benefit far outweighs this.

6. Feedforward Network (FFN): Per-Token Non-Linear Transformation

Attention handles "mixing between positions"; the FFN does non-linear transformation independently for each token — the main source of model capacity and knowledge storage (~2/3 of Transformer parameters reside in FFN). Modern implementations:

Classic FFN (with GELU activation):
  FFN(x) = W2 · GELU( W1·x + b1 ) + b2
  (intermediate dim = 4×d_model, e.g., d=4096 → intermediate 11008/14336)

SwiGLU variant (used by Llama, Qwen, Mixtral):
  FFN(x) = (W1·x ⊙ σ( W3·x )) · W2
  (σ is sigmoid; gating preserves "how much should activate" info, commonly 2/3·4d intermediate dim + small gains)

• GELU: a smoothed ReLU, preserving non-linearity with smoother gradients; the default since GPT-2.
• SwiGLU: gated linear unit, empirically outperforms GELU FFN under the same parameter budget, at the cost of one extra weight matrix (total parameters ~1/3 more). Widely adopted by modern LLMs.

7. Decoder-Only and Causal Masking: Why GPT Keeps Only Half

The original Transformer is encoder-decoder (encoder reads full sequence bidirectionally → decoder generates left-to-right). The GPT series made a decisive simplification: keep only the decoder's autoregressive branch, and use causal masking to hide future information — predicting position t only sees 1..t−1.

# Causal mask: upper triangle set to -inf, softmax makes future-position weights 0
n = seq_len
mask = torch.triu(torch.ones(n, n), diagonal=1).bool()   # upper triangle is True
# scores[i, j] is masked when j > i → position i can't see any tokens after it

Why almost all big models go decoder-only? Three reasons: first, natively generative — autoregressive objective aligns perfectly with "continuation," no need to train a separate encoder; second, unified training objective — every token participates in the loss, no "wasted" masked positions; third, few-shot learning — GPT-3 showed that pure decoder autoregressive pretraining is sufficient for in-context learning. See the GPT series and BERT and Encoder Families for the full route comparison.

Causal masking has another often-overlooked benefit: training and inference behavior are consistent. During training, the model naturally predicts each position step by step, and masking enforces the "position t only sees preceding context" constraint throughout; at inference time, it's naturally "already-generated prefix + new token" — no "bidirectional at training, unidirectional at inference" mismatch like encoder-style approaches have. This makes decoder-only the simplest and least error-prone engineering-wise, avoiding the subtle bugs of "training/deployment inconsistency."

8. Complexity and KV Cache

1. Complexity Analysis

Self-attention: for sequence length n and hidden dimension d:
  Computing Q·K^T: O(n²·d)     ← time and memory both grow quadratically with n
  Softmax and weighted sum: O(n²·d)
FFN: O(n·d²), linear in n

Conclusion: long-sequence cost is dominated by the O(n²) self-attention,
       which is the wall that long-context engineering (FlashAttention, sparse attention) must overcome.

More precisely: the matrix multiplication part can be greatly accelerated with low-precision and tiling, but the O(n²) "every pair of positions must interact" is structural — as long as attention is fully connected, doubling n quadruples the number of interactions. Three mitigation paths: constant optimization (FlashAttention), structural sparsity (local windows/global anchors), or fundamentally changing the mechanism (linear attention, state space models), detailed in Context & Long Context.

2. KV Cache: Reuse at Inference

During autoregressive decoding, each new token requires an attention computation, but the K and V of historical positions are exactly the same as before — no need to recompute. KV Cache stores and reuses them:

Without KV Cache: generating the t-th token requires re-forwarding 1..t entirely → O(t²) total cost
With KV Cache: only compute the new token's Q; K/V come straight from cache → per-step O(t), total O(t)
Cost: cache memory grows linearly with sequence length
  (≈ 2 × num_layers × num_heads × d_head × length × bytes_per_value)

Pseudocode (generation loop, illustrative):
for t in 1..max_len:
    q_t = last_hidden_t @ W_q
    k_t = last_hidden_t @ W_k; v_t = last_hidden_t @ W_v
    k_cache = cat(k_cache, k_t); v_cache = cat(v_cache, v_t)
    out_t = attention(q_t, k_cache, v_cache)   # weighted sum over full cache only
    ...

KV Cache is the primary variable for inference throughput and memory; its management and optimization (PagedAttention, quantization, sharing) are covered in Inference Fundamentals: Autoregression and Sampling and Deployment & Serving.

A memory number example (to grasp the magnitude): for a ~7B model (32 layers, 32 heads, d_head 128, bf16 = 2 bytes per value), a single 4096-token sequence's KV Cache ≈ 2 × 32 × 32 × 128 × 4096 × 2 bytes ≈ 2.1 GB; switching to 128K context → ~66 GB — exceeding most single-GPU memory. This is why long context inevitably requires KV Cache compression (quantization, sharing, paging) and multi-GPU deployment.

9. Why Transformer Is Suited for Large-Scale Parallelism

Transformer dominates LLMs not just because of quality, but because its computation shape is extremely friendly to distributed training:

Feature	Significance for Parallelism
No sequence dependency within layers	Tensor parallelism (splitting attention/FFN weights) has no communication hotspots
Per-token independent computation	Data parallelism (batch splitting) natively applies
Inter-layer only via residual connections	Pipeline parallelism (splitting by layer) efficiently overlaps computation and communication
Matrix-operation-heavy	High GPU utilization, can mix bf16 and other low-precision

This makes pretraining at the scale of thousands of GPUs (see Pretraining: Data and Objectives) possible, directly supporting the realization of scaling laws.

10. Three Architectural Forms Compared

Form	Structure	Bidirectional/Unidirectional	Representative Models	Strengths
Encoder-only	Encoder only	Bidirectional	BERT, RoBERTa	Understanding, retrieval, classification
Decoder-only	Decoder only (causal)	Unidirectional	GPT, Llama, Qwen, DeepSeek	Generation, few-shot, dialogue
Encoder-decoder	Encoder + Decoder	Mixed	Original Transformer, T5, BART	Translation, summarization, conversion tasks

One-line selection guide: want generation and dialogue → decoder-only; want representations and retrieval → encoder-only; want input understanding + output generation and can accept two-phase → encoder-decoder. In the big model era, decoder-only is default; the other two forms still have irreplaceable positions in specific tasks.

One-line summary

Every block in Transformer has a clear role: attention handles "who's related to whom", positional encoding handles "order", FFN handles "capacity", residual+Norm handles "stability", causal mask handles "generation", and KV Cache handles "fast inference". Understanding at this granularity means you truly understand the model.

11. Trade-offs and Boundaries

• O(n²) is the wall for long context: attention scales quadratically with sequence length; long text needs special engineering (see Context & Long Context).
• Positional encoding extrapolation is an inherent weakness: output quality degrades rapidly outside the training window — a known limitation to keep in mind during evaluation.
• Capacity relies on FFN stacking, relationships rely on attention learning: the allocation ratio between the two when scaling models directly affects downstream performance.
• Training stability > single-point tricks: Pre-Norm, scaling by √d_k, residual design all serve "stable large-scale training."

Attention Visualization and Debugging

"Where exactly is the model looking?" is a key entry point for understanding and troubleshooting: visualizing attention weights as heatmaps, common patterns include strong diagonal weights (each position mainly sees itself and nearby positions), noun-modifier edges (syntactic dependencies), head-of-sentence aggregation ([CLS]-style position absorbing global info). Debugging tips: when generation goes haywire, first check if attention is "flattening" (all positions have near-uniform weights, usually indicating undertraining or exceeding length); when certain tokens are consistently ignored, check if positional encoding matches the tokenizer. These tools (like attention rollout, visualization libraries) and the attention mechanism research in Core Paper Deep Dives are in the same lineage.

Further Reading

• Language Modeling: The Next-Token Prediction Paradigm — what objective the Transformer is training
• Pretraining: Data and Objectives — how the architecture is trained on trillions of tokens
• Context & Long Context — engineering solutions for O(n²) and positional encoding extrapolation
• Inference Fundamentals: Autoregression and Sampling — KV Cache and decoding strategies
• Scaling Laws — once the architecture is set, how scale determines capability
• Core Paper Deep Dives — deep guide to Attention Is All You Need

References

• Vaswani et al. Attention Is All You Need (2017) — the original Transformer paper
• Su et al. RoFormer: Enhanced Transformer with Rotary Position Embedding (2021) — the original RoPE paper
• Press et al. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (ALiBi, 2021) — ALiBi positional encoding
• Shazeer. GLU Variants Improve Transformer (2020) — the origin of SwiGLU
• Hendrycks & Gimpel. Gaussian Error Linear Units (GELU, 2016) — GELU activation function
• Touvron et al. LLaMA: Open and Efficient Foundation Language Models (2023) — public reference for modern Pre-Norm + RoPE + SwiGLU configuration
• Karpathy. nanoGPT — best code reference for building GPT from scratch