ML Handbook

Theme

Overfitting and Regularization

Quick overview Overfitting is machine learning's number one pitfall: the model mistakes noise in the training set for real patterns. This article explains the causes and signals of overfitting, plus six regularization weapons — L1/L2, Dropout, early stopping, data augmentation, pruning, ensembles — their principles, use cases, and tradeoffs.

Overfitting and Regularization ​

Concept Definition: Memorizing Answers ≠ Learning ​

Overfitting is the most classic and pervasive problem in machine learning: the model performs exceptionally well on the training data but poorly on new data. There's only one reason — the model memorized the noise in the training data as if it were a real pattern. It's like a student who memorizes the answers to practice exercises perfectly, but can't solve a differently phrased problem.

Two key insights for understanding overfitting:

  1. Performing well on the training set is inevitable — given sufficient capacity, a model can always fit the training data (even memorize every sample);
  2. Generalization (performing well on new data) is the only goal — memorizing the training set doesn't mean learning the pattern.

An extreme form of overfitting is "memorizing all training samples" (e.g., 1-NN with each point as a center). Regularization is the umbrella term for a set of techniques that "prevent the model from learning noise."

Causes and Signals of Overfitting ​

Three Causes ​

CauseMechanismTypical Scenario
Model too complexCapacity exceeds information provided by the data10-layer neural network on 100 samples
Too little / too uniform dataInsufficient information, model has to memorizeSmall samples + high-dimensional features
Noisy featuresMeaningless features give the model "room to play"High-dimensional sparse features, outliers

Signals to Watch For ​

The gap between training error and validation error is the golden signal:

  • Training loss keeps dropping, validation loss starts rising → overfitting has begun;
  • 99% training accuracy, 85% validation, with the gap widening → overfitting.

In the training curves of many frameworks (PyTorch, TF), the "V-shaped turning point" of validation loss is the critical point where the model switches from "learning patterns" to "memorizing noise."

The Bias-Variance Perspective ​

Overfitting = high variance, underfitting = high bias (see Model Evaluation and Validation). Regularization doesn't eliminate variance; it chooses between bias and variance — it introduces some bias (constraining model expressiveness) in exchange for a large drop in variance:

Error = Bias² + Variance + Noise
Strong regularization → Bias↑ Variance↓   (over-constrained: underfitting)
Weak regularization → Bias↓ Variance↑   (under-constrained: overfitting)
Optimum lies in between

This tradeoff is a unifying framework for understanding all regularization techniques — every form of regularization is essentially "taxing the model's complexity."

The Regularization Toolkit ​

1. L1 / L2 Regularization (Weight Penalty) ​

Add a penalty term on model weights in the loss function, keeping the model from "daring" to use large weights:

L2:  L = Original loss + λ·Σwᵢ²      (Ridge regression)
L1:  L = Original loss + λ·Σ|wᵢ|     (Lasso)
L2 (Ridge)L1 (Lasso)
PenaltySum of squared weightsSum of absolute weights
EffectWeights shrink overall (but rarely reach zero)Weight sparsification (many reach zero)
UseDefault regularization (smoothing)Feature selection (auto-selects features)
GeometryCircular constraint boundaryDiamond constraint boundary (vertices on axes → sparse)

Why L1 produces sparse solutions: L1's constraint region is a "diamond," and the optimum easily lands on a coordinate axis (one weight becomes exactly 0); L2's constraint region is a "circle," and the optimum generally lands on the circle rather than an axis. The practical value of sparsity: when there are many features, L1 automatically selects the useful ones, making the model simpler and more interpretable.

How to set λ (regularization strength): search on the validation set (see Hyperparameter Tuning). Too small λ has no effect, too large shrinks all weights to 0 (underfitting).

2. Early Stopping ​

Monitor validation loss during training, and stop training once validation loss stops decreasing (starts rising), taking the model at the validation optimum. Implemented with "save the best model + patience tolerance for a few fluctuations":

python
# Pseudocode: early stopping logic
best_val_loss, patience_counter = float('inf'), 0
for epoch in range(max_epochs):
    train_one_epoch()
    val_loss = evaluate(valid_set)
    if val_loss < best_val_loss:
        best_val_loss = val_loss
        patience_counter = 0
        save_checkpoint()              # save the best weights
    else:
        patience_counter += 1
        if patience_counter >= patience:  # no improvement for N consecutive rounds
            break

Early stopping is a "free lunch": it doesn't change the model structure, doesn't add computation, and directly cuts off overfitting. The default option for deep learning, almost always worth enabling.

3. Dropout (Deep Learning Specific) ​

During training, randomly drop the outputs of some neurons (set to zero with probability p), forcing the network to learn redundant representations where "no single neuron is indispensable." This is essentially simulating ensemble learning inside a single model — each forward pass is a different sub-network, and at inference time it averages them out.

  • Classic usage: p=0.5 for fully connected layers, p≈0.1 for convolutional layers;
  • Only enabled during training, must be disabled at inference time (or scaled by p);
  • Modern alternatives: newer versions of Dropout (DropBlock, Variational Dropout), plus the regularization effect of normalization layers (BatchNorm provides mild regularization).

4. Data Augmentation ​

Use prior knowledge to create more training samples, directly expanding the data volume — the most "fundamental" form of regularization. Images: random cropping, flipping, rotation, color jittering; Text: synonym replacement, back-translation; Tabular: SMOTE (synthetic minority samples), noise injection.

The fundamental assumption of data augmentation: these transformations don't change the semantic label (a flipped cat is still a cat). The more transformations match the real data distribution, the more effective. Its power endures in the era of large models: Vision Transformer papers repeatedly demonstrate the decisive role of data augmentation in data efficiency.

5. Model Simplification and Pruning ​

  • Structured simplification: reduce layers/neurons/tree depth, lower polynomial order — most direct but costs expressiveness;
  • Pruning: after training, cut unimportant weights/neurons, reducing model size while being able to fine-tune back to original accuracy;
  • The "indirect route" of ensemble methods: random forests / GBDT naturally resist overfitting through bagging / boosting (see Tree Models and Ensemble Learning).

6. Cross-Validation and Model Selection ​

Regularization strength itself is a hyperparameter — use cross-validation to pick λ/early stopping point, not gut feeling. Cross-validation also prevents "secondary overfitting during hyperparameter selection"; see Model Evaluation and Validation.

Practical Troubleshooting: Overfitting vs. Underfitting in Deep Learning ​

Typical combo for deep learning scenarios:

SymptomCheck FirstThen Act
Training loss not decreasingUnderfittingModel too small / wrong learning rate / data problem
Good training, poor validationOverfittingData augmentation → Dropout → Early stopping → L2 → Reduce model
Poor training and poor validationData/feature problemGo back to the data stage
Poor and volatile training/validationLearning rate / normalization problemAdjust learning rate, add normalization

Rule: first ensure the training set can be fitted (the model has capacity), then talk about preventing overfitting (adding regularization) — reverse the order and you'll be confused.

Regularization isn't "more is better"

Regularization is essentially "trading bias for variance." Over-regularizing will cause underfitting — if the model can't even learn the training set, forget generalization. The correct workflow: first let the model overfit on the training set (proving sufficient capacity), then gradually add regularization until the validation set is optimal.

Tradeoffs ​

  • L1 vs L2: use L1 when you want feature selection (many features, want sparse interpretability); use L2 as default overfit prevention; they can also be combined (Elastic Net).
  • Early stopping vs. Strong regularization: early stopping is nearly free, add it first; L2/Dropout add hyperparameter search space, add them later.
  • Data augmentation vs. Regularization: augmentation doesn't lose information (creates realistic samples), regularization loses expressiveness — augment first when possible.
  • Model complexity vs. Performance: simple models (linear + regularization) often beat complex models on clean, small datasets; the value of complex models shows when data volume and structural complexity are large.

Further Reading ​

References ​