Building a Deep Learning Project from Scratch

In one sentence: Building a deep learning project from scratch means connecting the concepts you've already learned (data, model, loss, optimizer, evaluation) into a runnable pipeline — from a bare machine to "a model that can be trained, evaluated, reproduced, and clearly explained to others" is the full journey.

Between paper concepts (see Neural Network Fundamentals and Backpropagation and Automatic Differentiation) and actually getting something to run, there are a lot of engineering details. This article walks through all the steps using a classic task — MNIST handwritten digit recognition (0–9, grayscale 28×28, 60,000 training images and 10,000 test images). Everything uses PyTorch, and the code is runnable as-is.

1. Goal and Environment Setup

1.1 Define the Project Goal First

Before any project starts, write down three sentences:

1. Task: Given a 28×28 grayscale handwritten digit image, output the class 0–9.
2. Success criteria: Test set accuracy > 99% (a LeNet-level CNN easily reaches this on MNIST; an MLP can get to 98%).
3. Boundaries: Skip data augmentation and model deployment for now; just get the pipeline working.

Goals determine resource allocation. MNIST is a "hello world" task — you can train it in minutes. Switching to CIFAR-10 or ImageNet means rethinking hardware and time (see Evaluation and Experiments).

1.2 Create an Isolated Environment

Dependency conflicts are frequent in deep learning. Never install to the global Python. Use venv or conda to isolate:

# Create a virtual environment
python -m venv .venv
# Activate (Linux/macOS)
source .venv/bin/activate
# Activate (Windows PowerShell)
.venv\Scripts\activate

# Confirm the interpreter is from the virtual environment
which python   # Linux/macOS
where python   # Windows

1.3 Install PyTorch

PyTorch's GPU support depends on CUDA, and installation commands vary strongly by machine configuration. The table below shows the stable versions at the time of writing — check the PyTorch website for the exact version:

Scenario	Installation command	Notes
No GPU (best for learning)	pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu	CPU version is small and has no compatibility issues
NVIDIA GPU + CUDA 12.4	pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124	Requires the matching CUDA driver on your machine
NVIDIA GPU + CUDA 12.1	pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121	Better compatibility with older GPUs
No GPU for a quick test	Install the CPU version and train right away (~1 minute per epoch on MNIST)	No GPU needed

Checking the installation

python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

torch.cuda.is_available() must be True for CUDA to be usable. Even if it's False, this article still works fine — CPU works too.

Dependencies: torch, torchvision (includes the MNIST dataset and common preprocessing), matplotlib (plotting), numpy. More details on training and experiment discipline can be found in Training Recipes and Hyperparameter Tuning.

2. Data Loading and Preprocessing

2.1 Download and Inspect the Data

torchvision provides MNIST directly. root specifies the cache directory, download=True downloads automatically:

from torchvision import datasets, transforms

# Shared for training/testing: convert to Tensor and normalize to [0, 1]
transform = transforms.Compose([
    transforms.ToTensor(),                 # Convert PIL/ndarray to a [0,1] Tensor
    transforms.Normalize((0.1307,), (0.3081,))  # MNIST mean / standard deviation
])

train_set = datasets.MNIST(root="./data", train=True, download=True, transform=transform)
test_set  = datasets.MNIST(root="./data", train=False, download=True, transform=transform)

2.2 "Look" at the Data Before Writing the Model

Before writing any model, print data shapes and plot some samples:

import matplotlib.pyplot as plt

x, y = train_set[0]
print(x.shape, y)              # torch.Size([1, 28, 28]) 3
fig, axes = plt.subplots(1, 5, figsize=(10, 3))
for i in range(5):
    img, label = train_set[i]
    axes[i].imshow(img.squeeze(0), cmap="gray")
    axes[i].set_title(f"label={label}")
    axes[i].axis("off")
plt.savefig("data_samples.png", dpi=150)

This step may seem redundant, but it immediately exposes the three most common data issues: channel order, value ranges, and label meanings. More details on data engineering can be found in Data and Data Engineering.

2.3 Why Normalization Matters

The difference between raw pixel ranges [0, 255] and [0, 1] is huge: neural networks prefer inputs near 0 with variance close to 1 (matching the scale of weight initialization). Otherwise, gradients get amplified or attenuated during propagation through deep layers (mechanism explained in Initialization and Normalization). MNIST's mean 0.1307 and standard deviation 0.3081 are official constants computed over the full training set — just look them up and use them.

2.4 DataLoader and Batching

from torch.utils.data import DataLoader

train_loader = DataLoader(train_set, batch_size=64, shuffle=True,  num_workers=2)
test_loader  = DataLoader(test_set,  batch_size=256, shuffle=False, num_workers=2)

Key points:

• shuffle=True is only for training — shuffling makes each batch i.i.d., preventing the model from learning batch order. No shuffling needed for the test set.
• batch_size is a trade-off between GPU memory usage and gradient noise. Its interplay with learning rate is covered in Training Recipes and Hyperparameter Tuning.

3. Model Definition: Starting with MLP

Don't jump to CNN right away. The goal of the first model is to verify that the "data → model → train → evaluate" pipeline works — even if it's just a simple Multi-Layer Perceptron (MLP):

import torch
import torch.nn as nn
import torch.nn.functional as F

class MLP(nn.Module):
    """Takes 28*28=784-dimensional input, two hidden layers, 10 output classes"""
    def __init__(self, input_dim=784, hidden=256, num_classes=10):
        super().__init__()
        self.fc1 = nn.Linear(input_dim, hidden)
        self.fc2 = nn.Linear(hidden, hidden)
        self.fc3 = nn.Linear(hidden, num_classes)

    def forward(self, x):
        x = x.view(x.size(0), -1)   # [B,1,28,28] -> [B,784], flatten
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)             # output logits, no softmax here
        return x

model = MLP()
print(model)

Two details worth noting:

• No softmax on the output layer: nn.CrossEntropyLoss internally applies LogSoftmax + NLLLoss; you only need to provide logits. Adding softmax beforehand causes numerical instability and double normalization (see Loss Functions and Output Layers).
• view flattening: MLPs don't understand 2D structure, so 28×28 is flattened into a 784-dimensional vector, discarding spatial relationships between pixels — which is exactly why you'll switch to CNNs later.

4. Training Loop: Write It Explicitly

Write out the full training loop — forward → loss → backward → update, nothing missing:

def train_one_epoch(model, loader, optimizer, criterion, device):
    model.train()                       # Switch to training mode (affects Dropout/BatchNorm)
    total_loss, correct, total = 0.0, 0, 0
    for x, y in loader:
        x, y = x.to(device), y.to(device)

        # 1. Forward pass: input -> predicted logits
        logits = model(x)
        # 2. Compute loss
        loss = criterion(logits, y)
        # 3. Backward pass: compute gradients for all parameters
        optimizer.zero_grad()
        loss.backward()
        # 4. Update parameters: one step down the gradient
        optimizer.step()

        total_loss += loss.item() * x.size(0)
        pred = logits.argmax(dim=1)
        correct += (pred == y).sum().item()
        total += y.size(0)
    return total_loss / total, correct / total

Why optimizer.zero_grad()? PyTorch accumulates gradients (loss.backward() adds to param.grad). Without zeroing, gradients from each batch would stack up, and the update direction would be wrong. This is one of the most common beginner traps (more traps in Common Pitfalls and Anti-Patterns).

Main loop and evaluation:

def evaluate(model, loader, criterion, device):
    model.eval()                        # Switch to evaluation mode
    total_loss, correct, total = 0.0, 0, 0
    with torch.no_grad():               # Don't track gradients — saves memory and time
        for x, y in loader:
            x, y = x.to(device), y.to(device)
            logits = model(x)
            loss = criterion(logits, y)
            total_loss += loss.item() * x.size(0)
            correct += (logits.argmax(1) == y).sum().item()
            total += y.size(0)
    return total_loss / total, correct / total

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

for epoch in range(10):
    train_loss, train_acc = train_one_epoch(model, train_loader, optimizer, criterion, device)
    test_loss, test_acc = evaluate(model, test_loader, criterion, device)
    print(f"epoch {epoch+1:2d} | train_loss {train_loss:.4f} | train_acc {train_acc:.4f} | test_acc {test_acc:.4f}")

Training / evaluation mode switching

model.train() and model.eval() must be paired. With the current model (no Dropout/BatchNorm) the impact is minimal, but once you add them (next section), forgetting to switch modes will cause weird bugs due to inconsistent training/inference behavior (mechanism explained in Common Pitfalls and Anti-Patterns).

On MNIST, this MLP trained for 10 epochs (a few minutes even on CPU) achieves about 98% test accuracy — first milestone complete.

5. Upgrading to CNN: Let the Model See Structure

The MLP's 98% is already decent, but its ceiling comes from treating images as vectors: it can't capture the fact that "adjacent pixels form strokes." Convolutional layers encode this prior through local receptive fields and weight sharing (principles in CNNs and Computer Vision):

class CNN(nn.Module):
    """Lightweight CNN in LeNet-5 style: convolutional feature extraction + fully connected classification"""
    def __init__(self, num_classes=10):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=3, padding=1),  # 28x28 -> 28x28
            nn.ReLU(),
            nn.MaxPool2d(2),                              # 28x28 -> 14x14
            nn.Conv2d(32, 64, kernel_size=3, padding=1),  # 14x14 -> 14x14
            nn.ReLU(),
            nn.MaxPool2d(2),                              # 14x14 -> 7x7
        )
        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(64 * 7 * 7, 128),
            nn.ReLU(),
            nn.Linear(128, num_classes),
        )

    def forward(self, x):
        return self.classifier(self.features(x))

model = CNN().to(device)

Only three changes, but they bring a fundamental shift:

• No manual flattening: nn.Conv2d takes [B,1,28,28] directly; Flatten only happens after feature extraction.
• Deeper channels: 32 → 64, with feature maps becoming progressively more abstract (edges → strokes → parts).
• Fewer parameters: This CNN has about 34K parameters, versus about 260K for the MLP above — weight sharing makes convolutions more efficient and more robust to overfitting.

With the same training code, the CNN reaches 99%+ test accuracy. Also try adding nn.Dropout(0.25) to the CNN to observe regularization effects (principles in Overfitting and Regularization).

6. Evaluation and Error Analysis

Accuracy is just the starting point. A proper evaluation answers: Which samples does the model get wrong? And why?

6.1 Confusion Matrix

from sklearn.metrics import confusion_matrix, classification_report
import numpy as np

all_pred, all_true = [], []
model.eval()
with torch.no_grad():
    for x, y in test_loader:
        logits = model(x.to(device))
        all_pred.append(logits.argmax(1).cpu().numpy())
        all_true.append(y.numpy())
all_pred = np.concatenate(all_pred)
all_true = np.concatenate(all_true)

cm = confusion_matrix(all_true, all_pred)
print(classification_report(all_true, all_pred, digits=4))

classification_report gives precision/recall/F1 per class. A common pattern on MNIST: confusion concentrates between visually similar digits like "4–9" and "3–8", and "9" often has slightly lower recall — such insights are far more useful than a single accuracy number, because errors in real-world applications tend to cluster heavily (evaluation design in Evaluation in Practice).

6.2 Visualize Misclassified Examples

mis_idx = np.where(all_true != all_pred)[0][:16]
fig, axes = plt.subplots(4, 4, figsize=(8, 8))
for i, idx in enumerate(mis_idx):
    img = test_set[idx][0].squeeze(0)
    axes[i // 4][i % 4].imshow(img, cmap="gray")
    axes[i // 4][i % 4].set_title(f"true={all_true[idx]} pred={all_pred[idx]}")
    axes[i // 4][i % 4].axis("off")
plt.suptitle("Misclassified samples")
plt.savefig("misclassified.png", dpi=150)

Looking at misclassified images usually reveals two categories of issues: data problems (mislabeling, blurry samples, extreme writing styles) and model blind spots (systematic failure on certain writing patterns). These require completely different fixes — the former calls for better data, the latter for model changes. More tools for interpretability analysis in Interpretability and Fairness.

7. Project Repository Structure

A project that others can understand and reproduce — structure matters more than code length:

mnist-project/
├── README.md            # Project overview, installation, run, results (template in [Portfolio Projects](/practice/portfolio-projects))
├── requirements.txt     # Pin dependencies: torch==2.x.x, torchvision==...
├── config.yaml          # All hyperparameters in one place (or argparse)
├── src/
│   ├── __init__.py
│   ├── data.py          # Dataset download, preprocessing, DataLoader
│   ├── model.py         # Model definition (MLP, CNN)
│   ├── train.py         # Training loop
│   ├── evaluate.py      # Evaluation, confusion matrix, misclassification visualization
│   └── utils.py         # Shared utilities: seed setting, device selection, etc.
├── notebooks/
│   └── explore.ipynb    # Data exploration and experiment logs
├── scripts/
│   └── run_experiments.sh  # One-command experiment reproduction
└── results/
    ├── checkpoints/     # Model weights
    └── figures/         # Training curves, confusion matrix, misclassification plots

::: tip Version pinning Use exact versions like torch==2.6.0inrequirements.txt, not torch>=2.0. Reproduction failures are rarely due to buggy code — they're usually from mismatched dependency versions. This is the core of "reproducible experiment discipline." :::

8. Common Pitfalls (This section is the most valuable)

Pitfall	Symptoms	Fix
Forgetting Normalize	Loss starts too high, slow convergence	Check input ranges, apply proper normalization
Forgetting optimizer.zero_grad()	Loss curve oscillates wildly, doesn't decrease	Zero gradients at the start of each batch
Forgetting model.eval() / no_grad()	Abnormal evaluation metrics, GPU memory bloat	Call model.eval() and wrap with torch.no_grad() before evaluation
Data leakage	Artificially high test accuracy, production collapse	Data augmentation and normalization statistics must be computed from training data only (see Evaluation in Practice)
Device mismatch	"Expected all tensors to be on the same device" error	Use x.to(device) and model.to(device) consistently
Tuning on the test set	Overfitting the test set, unreliable scores	Tune only on validation set; touch test set only once at the end

9. Next Steps

Once the pipeline works, branch out in three directions:

1. Harder datasets: CIFAR-10 (color 32×32, 50,000 training images) needs data augmentation + stronger CNN/ResNet. Training techniques are covered in Progressive Tutorial: Three Iterations.
2. Training tricks: Learning rate scheduling, BatchNorm, early stopping, mixed precision — each explained in Training Recipes and Hyperparameter Tuning.
3. Real deployment: Exporting, inference optimization, and production monitoring fall under MLOps — see MLOps and Model Deployment.

Final advice: Write down every pitfall you hit. The speed at which you encounter and reflect on pitfalls is the true rate at which your deep learning skills grow.

Further Reading

• Progressive Tutorial: Three Iterations — v1/v2/v3 iteration demo for the same task
• Training Recipes and Hyperparameter Tuning — default values and adjustment methods for every hyperparameter
• Debugging and Diagnostics — step-by-step diagnosis when a model doesn't work
• Datasets and Tools Reference — dataset directory beyond MNIST
• Curated Resources — more practical tools and resources
• MLOps and Model Deployment — the complete chain for getting a model into production

References

• LeCun, Bottou, Bengio, Haffner. Gradient-Based Learning Applied to Document Recognition (IEEE 1998) — Original LeNet-5 and MNIST paper
• Yann LeCun. THE MNIST DATABASE — MNIST dataset homepage
• PyTorch. PyTorch Documentation — Authoritative reference for all APIs used in this article