Deep Learning Handbook

Theme

Initialization and Normalization

Quick overview Weight initialization and normalization layers are the invisible prerequisite for deep networks that can actually be trained. This article explains why initialization matters, the intuition behind Xavier/He, compares BatchNorm/LayerNorm/GroupNorm/InstanceNorm for their applicable scenarios, and explains how the two complementarily tame gradients.

Initialization and Normalization ​

One-line definition: Initialization determines which "starting point" the network begins from, and normalization determines whether the "ground" is smooth to travel on — together, they ensure that gradient signals neither vanish nor explode in deep networks (root cause in Backpropagation and Automatic Differentiation), and they are the engineering prerequisite for "depth" to be viable.

I. Why Initialization Matters ​

Symmetry Breaking ​

If the weights of a layer are all initialized to 0, every neuron produces the same forward output at every step and receives the same backward gradient — they can never differentiate, and the network degrades to "one neuron" (symmetry was never broken). So initialization must be random, giving each neuron a different initial feature.

Gradient Scaling: Variance Conservation ​

A deeper reason is variance. During forward propagation:

z = W·x (assuming zero mean, independent), then Var(z) ≈ n_in · Var(w) · Var(x)

If Var(w) is too large, activations amplify layer by layer → forward explosion; the same happens in reverse, where gradients amplify/shrink layer by layer. To keep "variance conserved" in both forward and reverse passes, we need Var(w) ≈ 1/n_in (looking at input dimensions for forward) and ≈ 1/n_out (looking at output dimensions for backward) simultaneously.

One-sentence Memory Aid

The essence of initialization is variance control: keep the variance of every layer's input and output at O(1), so the forward pass doesn't explode and the backward pass doesn't vanish.

II. Xavier and He Initialization ​

Two of the most classic initialization rules, derived directly from "variance conservation":

MethodDistributionSuitable ActivationKey Idea
Xavier/Glorot (2010)Uniform or Gaussian, Var=2/(n_in+n_out)Sigmoid/Tanh (saturating activations)Simultaneously balance forward and backward variance
He/Kaiming (2015)Gaussian/uniform, Var=2/n_inReLU-family (non-saturating activations)ReLU zeroes the negative half, halving the information flow; variance must be doubled back

Why does He work for ReLU while Xavier doesn't? ReLU zeroes the negative half, so on average only half the information flow remains and the variance is halved. He's 2/n_in compensates exactly for this factor, ensuring forward variance conservation in ReLU networks. This is a classic case of "initialization must match the activation function," and it is the origin of the saying in Neural Networks Fundamentals that "activation, initialization, and normalization are three pieces that must be paired."

PyTorch usage:

python
import torch.nn as nn

def init_weights(m):
    if isinstance(m, nn.Linear):
        nn.init.kaiming_normal_(m.weight, mode='fan_in', nonlinearity='relu')
        if m.bias is not None:
            nn.init.zeros_(m.bias)
    elif isinstance(m, nn.Conv2d):
        nn.init.kaiming_normal_(m.weight, mode='fan_in', nonlinearity='relu')

model.apply(init_weights)

Subsequent residual connections (ResNet 2015) and normalization layers further relaxed the strict requirements on initialization — residuals let gradients "take a shortcut" along the identity path. See CNNs & Computer Vision.

III. Pretrained Initialization vs. Random Initialization ​

A second major shift in modern deep learning: start from something other than random initialization.

  • Random initialization: learn from scratch, requiring massive data and slow convergence.
  • Pretrained initialization: initialize with weights pretrained on a large corpus/image set (e.g., ImageNet-pretrained ResNet, wiki-pretrained BERT), then fine-tune on the target task.

Pretrained weights don't provide "parameters close to optimal" — they provide a good starting point for feature extraction — the model already "understands" edges, textures, and syntactic structures from the start. This "pretrain-finetune" paradigm is the core of Representation Learning and Pretraining.

Don't Destroy Pretrained Weights During Fine-tuning

Fine-tuning uses a small learning rate (10–100× smaller than for random initialization), and often layer-wise: shallow layers (learned general features) use a small LR or are frozen, while deep layers (task-specific) use a normal LR. Applying the large learning rate used for training from scratch will destroy the pretrained weights in one step.

IV. Normalization Layer Comparison ​

Normalization layers solve different versions of the same problem: keep the input of each layer within a controllable mean/variance range, eliminate internal covariate shift, and stabilize the gradient flow in a "smooth" region. Four mainstream normalizations are distinguished by "over which group of data the statistics are computed":

MethodNormalization DimensionTrain/Inference DifferenceApplicable Scenarios
BatchNorm (BN, 2015)By channel, across batch samplesTrain uses batch stats; inference uses running EMACNNs (works well when batch is large and stable)
LayerNorm (LN, 2016)By single sample, across channels/featuresConsistent train/inference, no stats issueTransformers, RNNs (small batch / variable length)
GroupNorm (GN, 2018)By single sample, grouping channels for normalizationConsistent train/inferenceSmall-batch CNNs (object detection)
InstanceNorm (IN, 2016)By single sample, per-channel normalizationConsistent train/inferenceStyle transfer, image generation

The core logic for choosing is "where are the statistics computed, and does it depend on the batch?"

  • BN depends on batch statistics: the smaller the batch, the more noise; performance degrades noticeably when batch_size < 16; it performs well with large batches in distributed training. Note: BN behaves differently at inference than at training (uses running averages), which is a common pitfall in multi-GPU / deployment scenarios.
  • LN is independent of batch: each sample is normalized independently, naturally suiting variable-length sequences (Transformers use LN instead of BN). This is one of the structural reasons Transformers chose LN (see Transformer Architecture).

PyTorch equivalents: nn.BatchNorm2d, nn.LayerNorm, nn.GroupNorm, nn.InstanceNorm2d.

V. The Complementary Relationship Between Normalization and Initialization ​

The two are not an either/or choice — they work together:

  • Initialization: ensures reasonable variance from "layer one," preventing explosion/vanishing at the initial stage.
  • Normalization: dynamically recalibrates at "every layer," keeping the input distribution stable even as parameters drift later.

With normalization layers, the tolerance for initialization is greatly broadened — normalization transfers the burden of "precisely controlling each layer's variance" from initialization to runtime statistics. Modern experience: normalization is more critical than initialization. In other words: if initialization is wrong, the model may not train at all; but as long as the correct normalization is used, the choice of initialization is much more forgiving (though Xavier/He remain the standard defaults). This complementary relationship extends further into optimization: normalization also mitigates sensitivity to the learning rate. See Optimization and Gradient Descent.

Fun Fact

BN also has a slight regularizing effect (the noise of batch statistics during training is equivalent to injecting perturbations), but don't treat BN as a regularizer — it was born for stabilizing training first. Real regularization techniques are covered in Overfitting and Regularization.

VI. Embedding and Positional Encoding Initialization ​

  • Embeddings (word vectors / image patch embeddings): default N(0, 1) small random initialization is fine, but note that embedding variance affects early training signals — a common practice in Transformers is to multiply embedding weights by √d_model or normalize after applying LayerNorm, to coordinate with the subsequent attention scaling.
  • Positional encodings: absolute positional encodings (sinusoidal or learnable) also need attention to scale. Learnable positional encodings use initialization with a standard deviation of about 0.02 (the GPT-series convention) to avoid magnitude imbalance with token embeddings. The three types of positional encodings (absolute/relative/RoPE) are covered in Attention Mechanisms.

VII. Common Pitfalls ​

  • BN and batch size: if GPU memory only allows a small batch when using BN → switch to GN or LN; don't push through.
  • BN behavior at inference: BN statistics come from different sources under model.eval() vs. model.train(). Forgetting to switch to eval() before evaluation leads to mysterious metric fluctuations — see Debugging and Diagnostics.
  • Freezing BN during fine-tuning: if you freeze preceding layers, the BN statistics (running stats) should be frozen together (requires_grad=False + momentum=0), otherwise the statistics drift.
  • Normalization order: in Transformers, Pre-LN (normalization before sublayers) is more stable than Post-LN and is the standard for large models. Getting the order wrong has a significant impact.
  • Initialization distribution type: the choice between uniform and Gaussian distributions has minimal impact — the key is to set the variance correctly according to the Xavier/He formulas; PyTorch's default initializations (kaiming_uniform, etc.) are typically already reasonable, so don't overwrite them arbitrarily.

VIII. Tradeoffs ​

Tradeoffs

BN's batch dependence vs. LN's sample independence: BN is fast and effective for large batches and CNNs; LN sacrifices cross-sample statistics (batch noise that brings regularization) in exchange for batch-independent stability. Rule of thumb: large-batch CNNs use BN, sequences/small-batch/variable-length use LN, small-batch CNNs use GN.

The benefit of normalization vs. compute/complexity: normalization layers add compute (albeit small) and inference state (BN's running stats); during deployment pruning and quantization, BN folding and LN simplification require dedicated handling (see MLOps and Model Deployment).

The benefit of pretrained initialization vs. flexibility: pretrained weights save data and converge faster, but they "anchor" the model to the structure of the pretrained task, making cross-domain transfer potentially less flexible than training from scratch. Additionally, pay attention to the licensing and privacy issues of pretrained weights.

Initialization and normalization are the "invisible foundation" — they don't directly boost representational power, but they determine whether the representational power of deep networks can be found by the optimizer. Get them right, pair them with the scheduling from Optimization and Gradient Descent and the defenses from Overfitting and Regularization, and you have a training pipeline that converges stably. For tuning details and diagnostics, see Training Recipes and Hyperparameter Tuning.

Further Reading ​

References ​