Optimization and Gradient Descent

Concept Definition: Training = Minimizing Loss

The essence of machine learning training is solving an optimization problem: find parameters θ that minimize the loss function L(θ). Taking linear regression as an example, L(w,b) = 1/n·Σ(yᵢ - ŷᵢ)², and training means gradually adjusting w and b along the "fastest direction of loss decrease" until the loss stops dropping.

This method of "walking in the fastest direction of decrease" is gradient descent — using the gradient (derivative vector) of the loss with respect to parameters to indicate direction, stepping in the negative gradient direction each time. The gradient is "the rate at which loss changes with parameters," and the negative gradient is "the fastest direction of decrease."

θ ← θ - η·∇L(θ)
      ↑    ↑
   Parameter  Learning rate (step size) × Gradient

Three Variants of Gradient Descent

Divided by "how much data is used to compute the gradient each time":

Variant	Used Per Update	Characteristics	Suitable For
Batch Gradient Descent (BGD)	All samples	Accurate gradient, but traverses the full dataset each step — slow, memory-intensive	Small datasets
Stochastic Gradient Descent (SGD)	1 sample	Fast per step, stochastic noise can escape local optima, but noisy and oscillating	Medium-large datasets
Mini-batch Gradient Descent	A small batch (32/64/128…)	Compromise: more accurate gradient + vectorized acceleration + controllable noise	The default for deep learning

Mini-batch is the practice standard: batch size is an important hyperparameter — too small means high noise and slow convergence, too large means accurate gradients but prone to sharp minima and high GPU memory consumption.

Why "noise" is a feature

SGD's noise lets it escape local minima (local optima); larger batches produce smoother gradients and more easily settle at sharp local minima (which often generalize worse). In practice, it's commonly observed that models trained with small batches generalize better. This is an instance of "noise as regularization."

Learning Rate: The Most Important Hyperparameter

The learning rate η determines how far to step each time — it is the first hyperparameter for almost every model:

Learning rate too large → Oscillation and divergence (loss rises / NaN)
Learning rate appropriate → Fast convergence
Learning rate too small → Snail's pace convergence, prone to being stuck in local optima

Learning Rate Scheduling

Fixed learning rate is rarely optimal. Common scheduling strategies:

Strategy	Approach	Characteristics
Step Decay	Multiply by 0.1 every N rounds	Simple, needs tuning of decay timing
Exponential Decay	Multiply by γ<1 each round	Smooth decay
Cosine Annealing	Cosine curve down to near 0	Fine convergence in late training, works well with warmup
Warmup + Decay	Linearly ramp up in the first few rounds, then decay	Standard for large models (prevents initial large gradients from destroying pre-trained weights)

How to Find the Initial Learning Rate

• Empirical ranges: start from 1e-3 (Adam) or 1e-2 (SGD+momentum) by default;
• Learning rate finder: run a few batches from very small with exponential growth, plot the loss-learning rate curve, and pick "the start of the steepest loss descent segment";
• Large model training: warmup to a peak in the 1e-4~1e-3 range.

Optimizers: From SGD to the Adam Family

Momentum: an improvement on SGD — the update direction adds the historical gradient direction (like a rolling ball accumulating momentum), accelerating convergence, suppressing oscillations, and allowing escape from local minima. Nesterov momentum further "looks one step ahead before moving."

Adaptive learning rate: each parameter gets its own learning rate, with frequently updated parameters getting small steps and sparse parameters getting large steps:

Optimizer	Core Mechanism	Characteristics	Suitable For
Adagrad	Cumulative gradient squared as denominator	Good for sparse features, but learning rate monotonically decays to zero	Sparse features
RMSProp	Sliding average of gradient squared	Fixes Adagrad's decay problem	General-purpose
Adam	Momentum + RMSProp (first- and second-order moment estimates)	Default choice, fast and stable convergence, nearly zero-tuning required	The deep learning default
AdamW	Adam + decoupled weight decay	More correct regularization, standard for large models/Transformers	Large models / Transformers
SGD+Momentum	Momentum + fixed learning rate	Often better generalization, but requires careful learning rate tuning	Small models, classic CV

Two things to keep in mind about Adam

1. Adam isn't a free lunch: it converges fast but often stops at "wider minima," and generalization can sometimes be worse than well-tuned SGD+Momentum;
2. Weight decay ≠ L2 regularization: in Adam, the gradient of L2 gets disrupted by second-order normalization. The correct approach is decoupled weight decay (AdamW) — essential knowledge for training Transformers.

Vanishing and Exploding Gradients

During backpropagation, gradients multiply layer by layer, and deep networks are highly prone to:

• Vanishing gradient: gradients approach 0, shallow-layer parameters barely update — deep networks are hard to train (the root cause before 2015);
• Exploding gradient: gradients grow exponentially — parameters diverge, loss becomes NaN.

Causes and Countermeasures

Cause	Countermeasure
Saturating activation functions (sigmoid/tanh have derivatives ≈ 0 at both ends)	Switch to ReLU/GELU and other non-saturating activations
Network too deep	Residual connections (ResNet), normalization, skip connections
Improper initialization	Xavier/He initialization, paired with activation functions
Learning rate too large	Reduce learning rate, use warmup
Exploding gradients	Gradient clipping: scale down if exceeding a threshold, standard for Transformer/LLM

Diagnostic method: print the gradient norm of each layer during training and check if they decrease/increase exponentially.

Loss Non-Convergence Troubleshooting Checklist

The most common troubleshooting scene in training: "loss stuck / jumping around / NaN," checked by priority:

1. Check data: NaN/infinite values? Messed-up labels? Shuffled?
2. Check learning rate: too large (NaN/oscillation) or too small (stuck)?
3. Check loss function: is the implementation correct? First try to overfit 1 batch to verify implementability
4. Check normalization: is input standardized? Unnormalized data is the #1 source of gradient issues
5. Check initialization: are Xavier/He used?
6. Check model: any bugs in forward/backward? Test by overfitting a single sample
7. Check gradients: print gradient norms, vanishing or exploding?

The fastest trick to verify model correctness

Try to overfit on a single batch (or even a single sample): if the model can't even fit one sample (loss doesn't drop near 0), it means there's a bug in the model/loss/data pipeline, not an optimization problem. This is the first move in deep learning debugging.

Interaction with Regularization

Optimization and regularization are a pair:

• Early stopping: monitor the validation set and stop training, itself a form of "implicit regularization" (limiting effective steps);
• Weight decay (L2): add a penalty term in the loss, equivalent to constraining parameter norm, and also stabilizes optimization;
• Noise as regularization: SGD's randomness, Dropout's random dropping, all inject noise into the optimization path in exchange for better generalization;
• Learning rate and generalization: overly high learning rate converges fast but the model "doesn't memorize" details well; small learning rates in late cosine decay often bring better generalization.

Tradeoffs

• Adam vs SGD: use Adam for quick deployment and peace of mind; use SGD+Momentum for optimal generalization when you have time to tune; use AdamW for large models/Transformers.
• Batch size: within GPU memory limits, large batches train faster but may sacrifice generalization; mitigate with learning rate warmup.
• Training speed vs convergence quality: cosine annealing + warmup sacrifices a bit of early speed for later accuracy — the fact-of-life standard for large model training.
• Automatic differentiation frameworks: PyTorch/TensorFlow have automated gradient computation; what you need to manage is these four knobs: learning rate, optimizer, scheduler, and clipping.

Further Reading

• Deep Learning Fundamentals — the mathematical mechanism of backpropagation
• Overfitting and Regularization — full discussion of early stopping, weight decay
• Hyperparameter Tuning — joint search of learning rate/batch size/optimizer
• Common Pitfalls and Anti-Patterns — classic loss non-convergence failure stories
• Math Primer — mathematical background of gradients, convex optimization

References

• Ruder. An overview of gradient descent optimization algorithms (2016/2017) — the authoritative survey of all optimizers
• Kingma & Ba. Adam: A Method for Stochastic Optimization (ICLR 2015) — original Adam paper
• Loshchilov & Hutter. Decoupled Weight Decay Regularization (AdamW, ICLR 2019) — original AdamW paper
• Smith. Cyclical Learning Rates for Training Neural Networks (2017) — original LR finder paper
• Glorot & Bengio. Understanding the difficulty of training deep feedforward neural networks (2010) — the foundation of initialization and gradient issues
• He et al. Delving Deep into Rectifiers (2015) — He initialization