DEEP LEARNING HANDBOOK
A systematic knowledge graph from data to intelligence — neural networks, backpropagation, attention, generative models, large language models, MLOps, and career paths.
50+ pages aren't a library to read cover to cover — they're a roadmap you can assemble on demand
Interviewing in 1–3 months? Focus on JD alignment, resume rewriting, and the interview question bank — every page targets the deep learning topics top companies test.
View path →Have time and want a solid foundation? Progress week by week from the guide to hands-on practice, with clear checkpoints each week.
View path →Already working? Skip the reading — jump to "question → page" for on-the-fly answers. Treat this as a reference manual.
View path →Hand-picked from each section — these are the ones you should read first
Answer "what is deep learning" in one sentence, a runnable NumPy code snippet, and three empirical milestones: how multi-layer neural networks automatically learn features from data, how it differs from classical machine learning, why it works, and its costs and boundaries.
GuideAttention is the heart of Transformers and all modern large models. This article clarifies the idea of "soft retrieval" along with QKV scaled dot-product attention, multi-head attention, three types of positional encoding, self-/cross-attention, and causal masking, while also covering engineering essentials like O(n²) complexity, FlashAttention, and KV cache.
Core ConceptsBackpropagation is the foundation of deep learning training: the chain rule + gradient backpropagation. This article starts from "why we need gradients," manually derives a small network step by step, then transitions to the two modes of automatic differentiation, and finally covers the correct usage of PyTorch autograd, common pitfalls, and the root cause of vanishing/exploding gradients.
Core ConceptsNeural networks are the foundation of all deep learning. This article starts from a single neuron, uses comparison tables to clarify six common activation functions, provides a PyTorch implementation of MLP forward propagation, and uses the universal approximation theorem to explain why depth is more efficient than width. Finally, it discusses the tradeoffs among capacity, depth, and data.
Core ConceptsGradient descent is the engine that makes neural networks "learn." This article starts from the learning rate intuition, traces the evolution of SGD, momentum, Adam/AdamW, covers learning rate scheduling, gradient clipping, and batch size scaling rules, and finally presents a ready-to-use engineering defaults table.
Core ConceptsThe nemesis of deep learning models is not "insufficient capability" but "memorizing too much." This article uses the bias-variance decomposition to clarify overfitting mechanisms, systematically covers L1/L2, Dropout, early stopping, data augmentation, label smoothing, EMA, Mixup, and other regularization weapons, and provides engineering experience on combination strategies and intensity selection.
Core ConceptsConvolutional Neural Networks were the first breakthrough that ignited deep learning. This article breaks down the three core building blocks—convolution, pooling, and residual connections—along with the architectural evolution from LeNet to ViT, covers the family of vision tasks, the transfer learning recipe (with PyTorch code) and training tricks, and surveys the current state of large vision models like CLIP, SAM, and VLMs.
Case StudiesDiffusion models sidestep the blurriness of VAEs and the instability of GANs through a "forward noise-adding, backward noise-removing" process, becoming the dominant engine for text-to-image, video, audio, and other generative AI tasks. This article unpacks DDPM principles, DDIM accelerated sampling, Classifier-Free Guidance, the Stable Diffusion stack, and LoRA/ControlNet, and discusses evaluation and limitations.
Case StudiesLarge Language Models are the culmination of Transformer scaling. This article covers pretraining and data composition, scaling laws, SFT and instruction fine-tuning, RLHF/DPO alignment, in-context learning and RAG, inference optimizations like KV cache, addresses hallucination and evaluation critically, and concludes with the leap from LLMs to Agents.
Case StudiesThe Transformer replaced recurrence with self-attention, becoming the de facto standard in the era of large models. This article breaks down the Encoder/Decoder architecture, multi-head self-attention, positional encoding, Pre-Norm/Post-Norm, provides training tricks like warmup, surveys the BERT vs GPT families and ViT/cross-modal extensions, and covers engineering optimizations like FlashAttention and KV cache.
Case StudiesClose-read a dozen+ classic papers that changed the field, one by one: Perceptron, Backpropagation, LeNet, AlexNet, ResNet, LSTM, Transformer, BERT, GPT-3, Adam, BatchNorm, GAN, DDPM. Each paper is broken down into four sections: background and problem, core method, key experiments, limitations and subsequent impact, with estimated reading time and linked pages.
Paper ReadsWhy deep learning learners should read papers: understand principles instead of memorizing APIs, stay on the frontier, and stand out in interviews. This page presents three reading paths (2-hour crash course / engineering deployment / research) and the five-page map of this section, along with the most important advice — take away mechanisms, not numbers.
Paper ReadsWalk through the complete deep learning pipeline with MNIST handwritten digit recognition: environment setup, data loading and preprocessing, model evolution from MLP to CNN, explicit training loops, evaluation and error analysis, project repository structure, and the pitfalls most beginners fall into. Ideal as your first end-to-end project.
PracticeA systematic troubleshooting approach when your model isn't working: start by verifying your pipeline with overfitting on a small batch, then work through a checklist for loss not decreasing, gradient inspection, NaN tracing, distribution visualization, unit testing, and ablation studies. Includes a high-frequency bug list to turn blind trial-and-error into methodical debugging.
PracticeDeep learning training is a blend of following proven recipes and exercising judgment in the moment. This article provides default recipe tables for data, initialization, loss, optimizer, learning rate, regularization, and normalization, explains learning rate range testing, batch size interplay, warmup and cosine annealing, transfer learning and memory management, and ends with experiment reproducibility discipline.
PracticeA categorized interview question bank for deep learning roles: 30+ questions across seven dimensions — fundamentals, architectures, training, math, engineering, LLMs, and open-ended questions. Each question includes answer key points and linked pages for targeted self-assessment aligned with your JD study plan.
CareerA quick-reference math guide for deep learning: linear algebra (vectors/matrices/norms/eigenvalues/SVD), calculus (derivatives/partial derivatives/chain rule/Jacobian/Taylor series), probability & statistics (distributions/expectation/Bayes/MLE/entropy & KL divergence), and optimization (convexity & gradient descent). Each formula includes an explanation of why it matters in deep learning.
ResourcesBrowse by section, or search directly
From the 1958 perceptron to the large-model era of 2026, a complete timeline of deep learning's three waves and two winters — clarifying how computation, data, and algorithms converged to drive each leap forward, and what the history teaches us today.
Deconstruct "a deep learning problem" into six stages — problem definition, data, model, loss, optimization & training, evaluation & deployment — with a training loop diagram and a "which section should I go to" navigation table for the entire site.
Clarify the conceptual boundaries and real division of labor among artificial intelligence, machine learning, deep learning, and classical algorithms — and use "data shape" as a practical ruler for making actionable selection judgments: when to use deep models and when to reach for GBDT.
With 54 pages of content, how do you use this site effectively? This page outlines three routes: a ~2-week job-hunting sprint, an ~8-week deep dive, and a "problem → page" desk-reference index — each with weekly pacing and acceptance criteria.
Answer "what is deep learning" in one sentence, a runnable NumPy code snippet, and three empirical milestones: how multi-layer neural networks automatically learn features from data, how it differs from classical machine learning, why it works, and its costs and boundaries.
Attention is the heart of Transformers and all modern large models. This article clarifies the idea of "soft retrieval" along with QKV scaled dot-product attention, multi-head attention, three types of positional encoding, self-/cross-attention, and causal masking, while also covering engineering essentials like O(n²) complexity, FlashAttention, and KV cache.
Backpropagation is the foundation of deep learning training: the chain rule + gradient backpropagation. This article starts from "why we need gradients," manually derives a small network step by step, then transitions to the two modes of automatic differentiation, and finally covers the correct usage of PyTorch autograd, common pitfalls, and the root cause of vanishing/exploding gradients.
"Models are the cups, data is the water" — data quality directly determines the ceiling of models. This article covers data pipelines, augmentation methods for three modalities, annotation and active learning, data quality governance (deduplication/debiasing/noise), data leakage defenses, distributed loading, and data version management.
Without reliable evaluation, deep learning projects cannot progress. This article covers train/val/test splits and the data leakage defense line, learning-curve diagnostics, classification/regeneration/generation metrics, hyperparameter search and experiment management, baseline discipline, and an analysis of the most common evaluation pitfalls.
Reinforcement learning enables agents to maximize cumulative rewards through trial and error; deep RL uses neural networks as its "brain." This article clarifies the MDP setup, the DQN family and Actor-Critic spectrum, the positioning of PPO/SAC, and extends to the exploration-exploitation trade-off, reward design, and the significance of RLHF for LLMs.
Discriminative models answer "what is this"; generative models answer "generate one." This article surveys the spectrum of explicit/implicit density models, covering VAE (ELBO/reparameterization), GAN (adversarial game/mode collapse), diffusion models (DDPM), and autoregressive generation, their principles and relationships, and provides a selection guide.
Weight initialization and normalization layers are the invisible prerequisite for deep networks that can actually be trained. This article explains why initialization matters, the intuition behind Xavier/He, compares BatchNorm/LayerNorm/GroupNorm/InstanceNorm for their applicable scenarios, and explains how the two complementarily tame gradients.
The stronger the model, the more we need to answer "why did it decide that, and is it fair to whom?" This article surveys post-hoc explanations (Saliency/Grad-CAM/LIME/SHAP), mechanistic interpretability (probes/circuits), the limitations of explanations and "explanation hallucinations," and expands into fairness bias, evaluation, and alignment safety.
The loss function is the source of gradient signals and the direct definition of "what the model optimizes for." This article explains why output layers and losses must match, covers classification (cross-entropy, BCE, label smoothing, focal), regression (MSE, MAE, Huber), ranking and contrastive losses, and provides guidance on multi-task balancing and debugging.
A model that works in the lab is still far from a reliable product — there's an entire engineering chain in between. This article covers the DL lifecycle panorama, experiment tracking and model registries, three deployment modes (online/batch/edge), inference optimizations (quantization/distillation/pruning), data and concept drift monitoring, CI/CD, and cost-latency trade-offs.
Neural networks are the foundation of all deep learning. This article starts from a single neuron, uses comparison tables to clarify six common activation functions, provides a PyTorch implementation of MLP forward propagation, and uses the universal approximation theorem to explain why depth is more efficient than width. Finally, it discusses the tradeoffs among capacity, depth, and data.
Gradient descent is the engine that makes neural networks "learn." This article starts from the learning rate intuition, traces the evolution of SGD, momentum, Adam/AdamW, covers learning rate scheduling, gradient clipping, and batch size scaling rules, and finally presents a ready-to-use engineering defaults table.
The nemesis of deep learning models is not "insufficient capability" but "memorizing too much." This article uses the bias-variance decomposition to clarify overfitting mechanisms, systematically covers L1/L2, Dropout, early stopping, data augmentation, label smoothing, EMA, Mixup, and other regularization weapons, and provides engineering experience on combination strategies and intensity selection.
The secret weapon of deep learning isn't just fitting ability — it's "learning good representations." This article clarifies feature hierarchies and end-to-end thinking, the semantic structure of embedding spaces, and surveys transfer learning and fine-tuning strategies, the pretrain-finetune paradigm, and goes deep into self-supervised learning (SimCLR/MAE) and representation evaluation methods.
Convolutional Neural Networks were the first breakthrough that ignited deep learning. This article breaks down the three core building blocks—convolution, pooling, and residual connections—along with the architectural evolution from LeNet to ViT, covers the family of vision tasks, the transfer learning recipe (with PyTorch code) and training tricks, and surveys the current state of large vision models like CLIP, SAM, and VLMs.
Recommender systems are the most profitable battlefield for deep learning in internet industry. This article covers the four-layer architecture of recall/ranking/re-ranking, from matrix factorization to neural collaborative filtering, two-tower recall, SASRec sequential recommendation, DCN/Wide&Deep CTR ranking, PinSage graph-based recommendation, and MMoE multi-objective learning, with practical perspectives on cold start, evaluation, and deployment.
Deep reinforcement learning combines neural networks with sequential decision-making, from DQN gaming and AlphaGo beating humans to RLHF/RLVR in LLMs, gradually entering the real world. This article unpacks DQN/AlphaGo principles, robot sim-to-real, game AI, and confronts real-world challenges of sample efficiency, safety, reward design, and distribution shift.
Diffusion models sidestep the blurriness of VAEs and the instability of GANs through a "forward noise-adding, backward noise-removing" process, becoming the dominant engine for text-to-image, video, audio, and other generative AI tasks. This article unpacks DDPM principles, DDIM accelerated sampling, Classifier-Free Guidance, the Stable Diffusion stack, and LoRA/ControlNet, and discusses evaluation and limitations.
Graph neural networks extend deep learning to structured data on nodes and edges. This article unpacks graph representations, the message passing framework, and three mainstream models — GCN, GAT, and GraphSAGE — covering node classification, link prediction, and graph classification tasks and applications (recommenders, molecules, physics simulation), while confronting limitations like neighbor sampling, over-smoothing, and scalability.
Large Language Models are the culmination of Transformer scaling. This article covers pretraining and data composition, scaling laws, SFT and instruction fine-tuning, RLHF/DPO alignment, in-context learning and RAG, inference optimizations like KV cache, addresses hallucination and evaluation critically, and concludes with the leap from LLMs to Agents.
Multimodal models enable AI to understand text, images, audio, and video simultaneously. This article unpacks contrastive alignment in CLIP/ALIGN, the "visual encoder + projector + LLM" architecture of LLaVA/Qwen-VL/GPT-4V, early/late/cross-attention fusion schemes, Gemini/Omni unified models, MMMU evaluation, multimodal Agents and RAG, and current limitations.
Recurrent Neural Networks are recursive architectures designed for sequential data—the foundation for language models, machine translation, and speech recognition. This article breaks down RNN unrolling, BPTT and vanishing gradients, the LSTM three-gate structure and GRU, compares them to Transformers, and reviews remaining use cases like streaming speech and online prediction.
Speech and audio are among the earliest and most deeply applied areas of deep learning. This article covers representations from waveforms/MFCCs/Mel-spectrograms, traces ASR evolution from CTC to Seq2Seq to Whisper's large-scale weak supervision, TTS from Tacotron/WaveNet to VITS and diffusion TTS, and introduces speech enhancement, separation, large audio models, and streaming latency engineering trade-offs.
The Transformer replaced recurrence with self-attention, becoming the de facto standard in the era of large models. This article breaks down the Encoder/Decoder architecture, multi-head self-attention, positional encoding, Pre-Norm/Post-Norm, provides training tricks like warmup, surveys the BERT vs GPT families and ViT/cross-modal extensions, and covers engineering optimizations like FlashAttention and KV cache.
VAEs and GANs are the two foundational paradigms of deep generative modeling: VAEs learn latent spaces through variational inference, GANs generate realistic data through adversarial games. This article breaks down the reparameterization trick and ELBO, minimax games and mode collapse, surveys the improvement lineage from DCGAN/WGAN-GP/StyleGAN and conditional generation, and compares the current state against diffusion models.
Close-read a dozen+ classic papers that changed the field, one by one: Perceptron, Backpropagation, LeNet, AlexNet, ResNet, LSTM, Transformer, BERT, GPT-3, Adam, BatchNorm, GAN, DDPM. Each paper is broken down into four sections: background and problem, core method, key experiments, limitations and subsequent impact, with estimated reading time and linked pages.
Important breakthroughs and trends in deep learning from recent years: scaling laws and synthetic data, MoE sparse experts, multimodal unification, RLHF and alignment (DPO/RLVR), diffusion application extension, test-time compute (o1-class), Agents and tool use, efficiency revolution, and scientific discovery. Each section covers what it is, why it matters, representative work, and linked pages, with guidance on how to read time-sensitive data.
Lay out deep learning's key papers across six decades into a single map organized by theme and timeline: foundational era, convolution era, sequence modeling, attention & large models, generative models, efficiency & optimization, and scientific applications. Each paper gets a one-sentence contribution, with a comprehensive table (year / paper / theme / contribution / linked pages).
Why deep learning learners should read papers: understand principles instead of memorizing APIs, stay on the frontier, and stand out in interviews. This page presents three reading paths (2-hour crash course / engineering deployment / research) and the five-page map of this section, along with the most important advice — take away mechanisms, not numbers.
Methodology FAQ for reading papers: how to choose your first paper, what to do when you can't understand, the "three questions" note-taking framework, how to judge paper quality (vs. baseline, ablation, open-source code), how to reproduce, and how to present a paper well in 5 steps, plus common pitfalls and reading discipline. Organized in Q&A format for on-demand lookup.
Paper reading routes organized by goal: Path A, a 2-hour crash course reading 3-5 classics; Path B, engineering deployment mastering BatchNorm, Adam, and fine-tuning papers; Path C, research advancement working through derivations and surveys. Each paper includes reading rationale, depth level, and linked pages, plus reading pace recommendations.
Walk through the complete deep learning pipeline with MNIST handwritten digit recognition: environment setup, data loading and preprocessing, model evolution from MLP to CNN, explicit training loops, evaluation and error analysis, project repository structure, and the pitfalls most beginners fall into. Ideal as your first end-to-end project.
The most common ways things go wrong in deep learning practice, with symptoms, causes, and fixes for each: data leakage, evaluation leakage, wrong learning rate, forgetting normalization, class imbalance, unfixed seeds, validation overfitting, BatchNorm traps, mixed precision bugs, fine-tuning learning rate too high, OOM, and the illusion of "looking correct."
A systematic troubleshooting approach when your model isn't working: start by verifying your pipeline with overfitting on a small batch, then work through a checklist for loss not decreasing, gradient inspection, NaN tracing, distribution visualization, unit testing, and ablation studies. Includes a high-frequency bug list to turn blind trial-and-error into methodical debugging.
Deep learning engineering isn't guesswork — it's a trainable set of mental habits. This article distills eight design principles: start with a simple baseline, data first, modularity and configurability, change one variable at a time, reproducibility, progressive complexity, default to suspecting your own code, and scalability thinking — with concrete actions for each.
Accuracy isn't the whole story. This article covers evaluation design in depth: choosing metrics based on business context, confidence intervals and multiple runs, A/B testing and online evaluation, failure analysis and error set review, evaluation set construction with anti-leakage discipline, human evaluation for generative models, and runnable code for a complete image classification evaluation pipeline.
Framework choice is not a matter of faith — it's about fit. This article compares PyTorch, JAX, and TensorFlow/Keras across ecosystem, usability, performance, and debugging, surveys multi-GPU training, inference acceleration, and HuggingFace ecosystem selection, and ends with a decision tree: "how to choose based on your scenario."
From "can run code" to "projects you can clearly explain": this article covers project selection criteria, a proven list of good projects (image classification, text classification, small LLM fine-tuning, diffusion fine-tuning, recommender systems), and end-to-end presentation methods — README templates, code quality, Gradio/HF Spaces deployment, turning projects into stories, and resume presentation techniques.
The same image classification task, iterated through three code versions: v1 is the minimal runnable MLP to get the pipeline going, v2 adds data augmentation, BatchNorm, learning rate scheduling, and early stopping, v3 switches to a pre-trained CNN and performs full evaluation. Each version clarifies "what problem it solves, what the trade-off is," with a comparison table.
Deep learning training is a blend of following proven recipes and exercising judgment in the moment. This article provides default recipe tables for data, initialization, loss, optimizer, learning rate, regularization, and normalization, explains learning rate range testing, batch size interplay, warmup and cosine annealing, transfer learning and memory management, and ends with experiment reproducibility discipline.
How to write a deep learning role resume that passes the initial screen: the "Background–Approach–Results–Reflection" four-part template for project experience with before/after rewrite examples, categorized skill section writing, six common resume mistakes, and role-specific resume focus areas.
A categorized interview question bank for deep learning roles: 30+ questions across seven dimensions — fundamentals, architectures, training, math, engineering, LLMs, and open-ended questions. Each question includes answer key points and linked pages for targeted self-assessment aligned with your JD study plan.
Map JD skill keywords to this handbook's pages and knowledge points: a 20+ row "skill keyword → page" mapping table, paired with a "can do / can't do / sort of" three-tier labeling system, to generate a personal study plan and study order.
A breakdown of 8 representative deep learning role JDs (domestic big-tech: CV/NLP/LLM/Algorithm; overseas: MLE/Research Scientist/Applied Scientist), extracting key responsibilities, hard requirements, bonus qualifications, and linked pages for each; includes a JD skill keyword frequency table and notes on salary and degree thresholds.
The entry point for the career section: four steps of "see the market → see yourself → package yourself → prove yourself" tie together five articles; covers responsibilities, core skills, and broad salary references for nine types of DL roles; provides a three-dimensional decision checklist to help you choose a direction.
A curated collection of deep learning learning resources: classic courses like Andrew Ng's Deep Learning Specialization, CS231n, CS224n, Karpathy's Zero to Hero; essential books like the Deep Learning "Bible" and D2L; high-quality blogs like distill.pub and Lil'Log; tool ecosystems like PyTorch and HuggingFace; and conferences and paper collections.
A deep learning dataset archive: images (MNIST/CIFAR/ImageNet/COCO), text (GLUE/SQuAD/Wikipedia/The Pile), speech, multimodal, and molecular/graph datasets, with size/task/source/notes for each. Includes one-line profiles for PyTorch, HuggingFace, peft, and other tools, plus decision principles for choosing datasets.
A quick-reference dictionary of core deep learning terms: seven groups of 60+ terms covering fundamentals, models & layers, training & optimization, regularization, generative models & LLMs, evaluation & data, and production & trust. Includes distinctions of commonly confused terms and deep-dive links throughout the handbook.
A quick-reference math guide for deep learning: linear algebra (vectors/matrices/norms/eigenvalues/SVD), calculus (derivatives/partial derivatives/chain rule/Jacobian/Taylor series), probability & statistics (distributions/expectation/Bayes/MLE/entropy & KL divergence), and optimization (convexity & gradient descent). Each formula includes an explanation of why it matters in deep learning.
