Debugging and Diagnosis

One-sentence definition: debugging deep models comes down to building a "layered chain of evidence" — first prove the data pipeline is fine, then prove the gradients are fine, and only then suspect the model design. The vast majority of "model won't train" problems live in the first two layers.

A harsh reality of deep learning is that getting training to "run" (no errors) and getting training to be "correct" are two different things. A loss that decreases might be driven by data leakage; a loss that doesn't decrease might be a code bug rather than insufficient model capacity. This article provides a step-by-step troubleshooting workflow. For the statistical meaning of diagnostic signals, see Evaluation and Experiments and Backpropagation and Automatic Differentiation.

Rule #1: Overfit to 100% on a Small Subset First

The first test for any new model or data pipeline is: take 5–50 samples (typically one batch) and train until training loss approaches 0 and training accuracy hits 100%. This is the cornerstone of the entire debugging methodology.

import torch, torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset

# Take the first 32 samples from the training set
subset = torch.utils.data.Subset(train_set, list(range(32)))
loader = DataLoader(subset, batch_size=32, shuffle=True)

model = YourModel()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for epoch in range(100):
    model.train()
    for x, y in loader:
        optimizer.zero_grad()
        loss = criterion(model(x), y)
        loss.backward()
        optimizer.step()
    print(epoch, f"loss={loss.item():.4f}")

Success criterion: training loss should reliably drop near 0 (on the order of 0.001 for classification tasks).

• If you can't: there's a bug in model capacity, data shapes, or the forward/loss/gradient chain — this is the easiest stage to locate, since you only have 32 samples and no randomness to obscure things.
• If you can: congratulations, the pipeline and gradient chain are basically correct. Now you move on to "generalization problems" (overfitting/underfitting), which falls under Training Recipes and Hyperparameter Tuning.

Don't skip this step

Many people jump straight into full-batch training, run for 20 minutes, discover "loss isn't going down," and then start suspecting all kinds of mysterious causes. The small-subset overfitting test compresses the same troubleshooting into 30 seconds — it's the first step of every deep learning debug session.

Loss Not Decreasing: Troubleshooting Checklist

Assuming the small-subset test fails, check items in order of priority:

Priority	Suspect	Quick Verification
1	Wrong labels	Print x.shape, range and type of y; visualize to manually check labels (see Build Your Own DL Project)
2	Data not normalized / abnormal value range	Check x.min() / x.max(); image inputs should be in [0,1] or near 0 after standardization
3	Output layer mismatched with loss	Use logits + CrossEntropyLoss for classification, don't apply softmax twice; see Loss Functions and Output Layers
4	Wrong learning rate	Range testing: too large causes divergence, too small makes loss crawl almost imperceptibly
5	Gradients not being applied	Compare param.grad before and after loss.backward(); check if you forgot optimizer.step() / zero_grad()
6	Random baseline is too high	For 10-class classification, initial loss should be ~ln(10) ≈ 2.30; significant deviation indicates initialization or data issues
7	Input/output shape mismatch	Print the shape of every layer's output to locate the first layer that doesn't match expectations

# Quick check that gradients are actually being updated
for name, p in model.named_parameters():
    if p.requires_grad:
        print(name, "grad_ok" if p.grad is not None and p.grad.abs().max() > 0 else "NO_GRAD")

Gradient Checking (gradcheck)

If your model has custom operators or hand-written forward/backward passes, compare numerical gradients against analytical gradients. PyTorch has torch.autograd.gradcheck built in:

from torch.autograd import gradcheck

# Wrap as a function that takes (x,) as input (test on a single sample, typically in double precision)
model_double = MyCustomOp().double()
x = torch.randn(1, 16, dtype=torch.double, requires_grad=True)

# gradcheck approximates gradients via numerical differencing and compares them to autograd results
assert gradcheck(lambda inp: model_double(inp), (x,), eps=1e-6, atol=1e-4)
print("gradcheck passed")

Usage notes: use double precision (numerical differencing has too much error in float32); single sample or very small batch. For standard operators inside nn.Sequential, PyTorch has already verified them — no need to recheck. Reserve gradcheck for custom layers, hand-written losses, and self-implemented backward passes.

NaN / Inf Troubleshooting

When loss turns nan mid-training, check from most to least likely:

Cause	Mechanism	Fix
Learning rate too large	Parameters diverge, intermediate values overflow	Lower learning rate; range testing to find the right interval
Division by zero / log(0)	Zero denominator in normalization layers; log(0) in NLL	Add eps to denominators; avoid extreme values in data
Gradient explosion	Deep chain-rule multiplication amplifies errors (see Backpropagation)	Gradient clipping with clip_grad_norm_
Mixed precision (AMP) overflow	float16 upper bound is ~65504; small values underflow / large values overflow	Use GradScaler; check if custom operators support fp16
Data itself contains NaN	Upstream data cleaning missed them	Check inputs with torch.isnan(x).any()
Learning rate scheduler division by zero	Incorrect scheduler step argument	Check LambdaLR / ReduceLROnPlateau parameters

A general technique to locate "at which step the first NaN appears" is binary replay: record the loss for each batch, find the first batch where NaN appears, then reproduce that single batch + the model state at that point in isolation.

# Gradient clipping (a universal safety net for NaN, but only masks upstream problems — not a cure)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

The Visualization Diagnostic Trio

Numbers have limits; visuals don't. Three high-frequency visualization techniques can diagnose most problems:

1. Training Curves (loss / accuracy vs. steps)

• Both training and validation loss fail to go down → underfitting; increase capacity or adjust learning rate.
• Training loss goes down, validation loss stalls or rises → overfitting; add regularization (see Overfitting and Regularization).
• Loss oscillates violently without converging → learning rate too large or batch too small.
• Loss drops precipitously then plateaus → might be crossing a "narrow valley" in the loss landscape; try a different optimizer or scheduler.

2. Weight and Activation Distributions

Check whether the model's internal states are healthy using histograms:

import matplotlib.pyplot as plt

# Attach a forward hook to grab activations from a specific layer
activations = {}
def hook_fn(name):
    def hook(module, inp, out):
        activations[name] = out.detach().flatten().cpu()
    return hook
model.layer2.register_forward_hook(hook_fn("layer2"))

# After training for a few steps
plt.hist(activations["layer2"].numpy(), bins=100)
plt.title("activation distribution of layer2")
plt.savefig("activations.png")

Healthy signals: activation distributions should not collapse (no large constant regions), not saturate (for ReLU, the dead-region ratio should not be excessively high). A large number of neurons consistently at zero indicates "dead ReLU," typically caused by a learning rate that is too large or improper initialization (see Initialization and Normalization). Weights with exploding magnitudes are a precursor to gradient explosion.

3. Gradient Norms

Plot p.grad.norm() for each parameter group, watching for signs of explosion (magnitude spiking) or vanishing (approaching zero):

total_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=float("inf"))
print(f"grad norm: {total_norm:.4f}")

Unit Testing: Turn Known Bugs into Permanent Guardrails

Debugging experience should be hardened into regression tests. Three dimensions worth testing:

# 1) Shape and dimensionality
def test_output_shape():
    model = YourModel()
    out = model(torch.randn(4, 3, 32, 32))   # batch=4, CIFAR-10 input
    assert out.shape == (4, 10)

# 2) Masking / attention: masked positions should not contribute
def test_mask_ignored():
    x = torch.randn(2, 5, 16)
    mask = torch.tensor([[1, 1, 1, 0, 0], [1, 1, 0, 0, 0]])
    out_masked = YourModel()(x, mask=mask)
    # Expected: masked positions do not participate in aggregation, result should match
    out_ref = YourModel()(x * mask.unsqueeze(-1), mask=None)
    assert torch.allclose(out_masked, out_ref, atol=1e-4)

# 3) Training can advance: loss should decrease after one step (on fixed input)
def test_loss_decreases():
    model = YourModel()
    opt = torch.optim.SGD(model.parameters(), lr=0.01)
    x, y = fixed_batch()                      # fixed data and seed
    loss0 = model.loss(model(x), y)
    opt.zero_grad(); loss0.backward(); opt.step()
    loss1 = model.loss(model(x), y)
    assert loss1 < loss0

Organize with pytest. Write a test case for every pitfall you've encountered. Tests are not a luxury for post-production; they are the hardening of your debugging methodology.

Ablation Studies to Locate "Which Layer is Broken"

When the whole system runs but metrics don't meet expectations, use ablation to turn off components one by one and measure their contribution. Principle: change only one variable at a time.

A typical ablation matrix (image classification):

Config	Validation Accuracy	Conclusion
Baseline (augmentation + BN + cosine annealing)	78%	—
Remove data augmentation	70%	Augmentation contributed 8 points
Remove BatchNorm	74%	BN contributed 4 points
Remove cosine annealing (constant lr)	75%	Scheduler contributed 3 points

If removing a component makes things better, don't panic — this is still useful information. It means that component is mismatched with your data/model configuration, warranting deeper investigation, rather than being a "must-have" default. Ablation scales linearly in training cost, so think carefully about which variable is most suspicious and offers the biggest payoff.

Common Bug List (sorted by frequency)

Bug	Symptom	One-line Fix
Forgot optimizer.zero_grad()	Loss oscillates wildly	Zero grads before every batch
Forgot model.eval() / torch.no_grad()	Validation metrics are garbage, VRAM keeps growing	Switch modes for evaluation
Train/inference mode mismatch (BatchNorm/Dropout)	Looks great in training, collapses at inference	Check eval() timing
Device mismatch	Expected all tensors to be on the same device	Use consistent .to(device)
Wrong normalization stats (see Common Pitfalls and Anti-patterns)	Results are biased but no errors	Check where mean/std come from
Data leakage (augmenting stats from test set)	Inflated metrics	Only fit stats on training set
Hand-coded gradient errors	Training doesn't converge but no error messages	Use gradcheck
Custom loss overflows in mixed precision fp16	Intermittent NaN	Switch to fp32 or fix the operator
Forgot optimizer.step() after loss.backward()	Loss stays flat	Check the four-step loop
Wrong indexing/slicing causes mask misalignment	Weird results	Unit-test masking logic

Debugging mindset

Change one variable at a time, record a baseline before each change, and preserve evidence (logs / plots / numbers) for each modification. "I changed a few things and now it works" is unattributable — that's not debugging at all. For mindset and methodology in depth, see DL Design Principles.

Further Reading

• Build Your Own DL Project — the carrier project for debugging
• Training Recipes and Hyperparameter Tuning — recipe-side checks when loss doesn't go down
• Common Pitfalls and Anti-patterns — expanded version of this article's bug list
• DL Design Principles — meta-rules for ablation and experimental discipline
• Backpropagation and Automatic Differentiation — theory of the gradient chain
• Initialization and Normalization — theoretical origins of weight/activation distribution anomalies

References

• Karpathy. A Recipe for Training Neural Networks — the original source for the "overfit a batch first" methodology
• PyTorch. torch.autograd.gradcheck — official documentation for gradient checking
• Schoenholz et al. Deep Information Propagation through Nonlinearities (ICLR 2017) — analysis of activation distributions and network trainability