Diffusion Models and Generative AI

A diffusion model is a generative model that learns the data distribution — and generates new samples — by "adding noise step by step in the forward pass, denoising step by step in the reverse pass." It is the outright dominant technology in today's text-to-image and text-to-video generation. Stable Diffusion, open-sourced in August 2022, made "type a sentence, get an image" an everyday experience for ordinary people; in 2024, OpenAI's Sora extended that to "type a sentence, get a video." From Midjourney to DALL·E 3, from Imagen to Kling, nearly every first-tier image/video generation model runs on this machinery.

Why did diffusion models dethrone GANs, the former king? In one line: more stable training, higher quality, stronger controllability. It doesn't rely on adversarial training where two networks battle each other; instead it turns "generation" into a plain regression problem — predicting noise. And its sampling process naturally accepts all kinds of conditioning (text, images, poses, depth), making "controllable generation" possible. This page starts from the core principles and works through "pixel space → latent space → conditioning → applications and risks." To build the big picture of generative AI first, read What Are the Hot AI Concepts; to see where this fits in the full stack, check Anatomy of the Overall Architecture.

1. The Core Idea: Playing Destruction in Reverse

The idea comes from statistical physics, first proposed by Sohl-Dickstein et al. in 2015; the 2020 Denoising Diffusion Probabilistic Models (DDPM) paper by Ho et al. turned it into a practical tool that crushed GANs.

The core insight fits in one metaphor: it's easy to noise a photo step by step until it's unrecognizable — so can we train a network to learn to play that process in reverse?

Forward process (destruction — done during training, nearly free)
  real image x₀ ──add noise──▶ x₁ ──add noise──▶ … ──add noise──▶ x_T ≈ pure Gaussian noise

Reverse process (reconstruction — what the model must learn)
  pure noise x_T ──denoise──▶ x_{T-1} ──denoise──▶ … ──denoise──▶ brand-new image x₀

Three key properties make the idea work:

1. Forward destruction is "easy to obtain": the noised image at any timestep can be sampled in closed form from the original in one shot (see Section 2) — no step-by-step iteration needed during training;
2. Forward destruction is "easy to invert": the exact inverse of one noising step is unknown, but you can train a neural network to predict "what noise got mixed in" and thereby play the destruction in reverse;
3. Generation is denoising: start from pure noise and denoise step by step, and the result naturally follows the training data distribution P(x) — this is "learn the data distribution, and you can generate new samples."

The mathematical foundation here is the score-matching view: what the denoising network really learns is the gradient of the data's log-density ∇_x log p(x); starting from any noise and following that gradient eventually lands you on the data manifold. For the essential differences among the three mainstream generative routes (VAE/GAN/diffusion), see Section 5.

2. Core Principles: DDPM's Mathematical Skeleton

1. Forward Noising: A Closed-Form Solution

Let x₀ be a real image; a noise schedule defines the per-step variance β₁, β₂, …, β_T (DDPM uses T=1000, with β rising linearly from 1e-4 to 0.02). The forward process is a Markov chain:

q(x_t | x_{t-1}) = N( x_t ; √(1−β_t)·x_{t-1}, β_t·I )

Thanks to the additivity of Gaussians, the state at step t needs no iterative stepping — it can be written directly from x₀ in one shot:

x_t = √(ᾱ_t)·x₀ + √(1−ᾱ_t)·ε,   ε ~ N(0, I)

where α_t = 1 − β_t and ᾱ_t = ∏_{s=1}^{t} α_s. Picture the equation:

x_t = √(ᾱ_t) · x₀ + √(1−ᾱ_t) · ε
      └─image part─┘   └────noise part────┘
      t=0: pure image    t=T: pure noise (ᾱ_T→0)

√(ᾱ_t) controls "how much original image remains"; √(1−ᾱ_t) controls "how much noise has been mixed in." When T is large enough, ᾱ_T → 0 and x_T converges to standard Gaussian noise N(0, I). During training, draw a random t per sample and apply the formula to get a noised image instantly — one reason diffusion training is "cheap."

2. Reverse Denoising: The Training Objective Is "Predict the Noise"

We train a neural network ε_θ (backbone: UNet or DiT — see Section 3) to predict the noise ε mixed into x_t. DDPM's simplified training objective is plain mean squared error (MSE):

L = E_{t, x₀, ε} [ ‖ ε − ε_θ( √(ᾱ_t)·x₀ + √(1−ᾱ_t)·ε , t ) ‖² ]

In plain words: pick a random timestep t, add the noise, and have the network predict "the noise that was mixed in" from "the noised image + timestep t," measured by MSE. The network also receives t (a timestep embedding) as input, because the required "denoising aggressiveness" changes over time — the larger t, the more noise and the bolder the denoising.

Why "predict the noise" instead of "predict the image" directly?

The two are mathematically equivalent: knowing x_t and ε lets you recover x₀. But experiments found that the "predict the noise" signal trains more stably with lower variance — DDPM's key engineering improvement over earlier diffusion models. For the deeper score-matching / SDE view, read Song et al.'s unified framework (arXiv:2011.13456), listed in the References.

3. Sampling: Step-by-Step DDPM and DDIM Acceleration

Once training is done, generation means starting from x_T ~ N(0,I) and repeatedly applying the reverse step:

x_{t-1} = (1/√α_t)·( x_t − (β_t/√(1−ᾱ_t))·ε_θ(x_t, t) ) + σ_t·z,  z ~ N(0, I)
          └── strip the predicted noise, recover a cleaner state ──┘   └─random jitter─┘

σ_t is the per-step random jitter that keeps the reverse process a true inverse of diffusion. Watch a sampling run with your own eyes: the first dozens of steps pull outlines out of chaotic noise, the last dozens refine texture — so num_inference_steps (the number of sampling steps) directly determines both quality and runtime.

DDIM (Denoising Diffusion Implicit Models, 2021) rewrote the sampling trajectory by constructing a non-Markovian reverse process, enabling two things:

• Deterministic sampling: set the random jitter to zero and the same noise seed always yields the same image — the foundation of img2img, inpainting, and related features;
• Big jumps: sample only at 20–50 selected timesteps with little quality loss.

The one-line verdict: DDPM is the theoretical foundation; DDIM was the first mass-produced accelerator that pushed text-to-image from "one image a minute" to "one image every few seconds." The full skeleton of training and sampling fits in under twenty lines of pseudocode:

# Training (pseudocode)
for x₀ in train_batch:                # grab a real image
    t ~ Uniform({1, ..., T})          # sample a timestep (T=1000)
    ε ~ N(0, I)                       # sample random noise
    x_t = √ᾱ_t * x₀ + √(1−ᾱ_t) * ε   # one-shot closed-form forward noising
    loss = MSE(ε_θ(x_t, t), ε)        # the network predicts the noise
    optimizer.step(loss)

# Sampling (pseudocode)
x_T ~ N(0, I)                         # start from pure Gaussian noise
for t in reversed(timestep_sequence):   # DDPM: T→1; DDIM: only 20–50 steps
    ε_pred = ε_θ(x_t, t)              # the network predicts the current noise
    x_{t-1} = denoise_update(x_t, ε_pred, t)  # denoise + random jitter (DDIM: zero it out)
return x₀                             # the generated image

3. From Pixel Space to Latent Space: Latent Diffusion and Stable Diffusion

DDPM has a fatal flaw: it's slow. Iterating 1000 steps in a 512×512×3 pixel space makes training and inference too expensive for commercial use. The Latent Diffusion Model (LDM, 2022) dissolved the problem with one move: don't diffuse in pixel space — diffuse in a VAE latent space first. This is the direct ancestor of Stable Diffusion.

1. Why the Latent Space

Images are highly redundant: neighboring pixels are strongly correlated, and the true semantic information is far less than the pixel count. Compress the image into a low-dimensional latent space with a Variational Autoencoder (VAE) first, then diffuse — compute drops by roughly two orders of magnitude:

	Pixel-space diffusion (DDPM)	Latent-space diffusion (LDM/SD)
Working dimensions	512×512×3 ≈ 780K dims	64×64×4 ≈ 16K dims
Relative compute	~1	~1/50
Generation resolution	Struggles beyond 128×128	512×512 and 1024×1024 with ease

Better still, the VAE's latent space is a semantically meaningful continuous space: interpolating, blending, and editing in latent space translates into smooth, natural changes in the image — the stage on which all of Section 4's conditioning techniques perform.

2. Architecture: Text Encoder + UNet/DiT + VAE

Stable Diffusion's generation pipeline is a collaboration among three modules; here it is as a text diagram:

                    ┌──────────────────────────────────────────┐
  text prompt ──▶ CLIP text encoder ──▶ text embeddings (77 tokens × 768 dims)
   "a red fox..."                                        │
                                                         ▼
  random noise z_T ──▶ UNet/DiT (latent denoising) ──▶ latent z₀ ──▶ VAE decoder ──▶ image
                        ▲  ▲                                  │
                        │  └── cross-attention injects text conditioning
                        └────── timestep embedding (current step t)

Division of labor:

Component	Role	Notes
VAE encoder	Image → latent space (8× compression)	Used to build training samples during training
VAE decoder	Latent → pixel image	The final step of generation
UNet / DiT	Predicts noise in latent space (step-by-step denoising)	The model's "brain"; SD 1.5's UNet has ~860M parameters
CLIP text encoder	Prompt → text embeddings	SD 1.x uses CLIP ViT-L/14; SDXL uses a dual encoder
Cross-Attention	Injects text conditioning into every layer of the denoising network	The core mechanism behind text-to-image

UNet is an encoder–decoder convolutional network with skip connections: the encoder downsamples layer by layer to extract multi-scale features, the decoder upsamples to reconstruct, and skip connections preserve detail. It was first proven in medical image segmentation, then adapted for diffusion: timestep embeddings added at every layer, plus cross-attention to text in the middle layers.

Cross-attention is where the text-to-image magic happens: UNet's intermediate features serve as the Query and the text embeddings as Key/Value, so "every position in the image attends to the words relevant to it." This is the idea of Transformers and Attention transplanted into image generation — the first time images and text became deeply coupled at the feature level.

Backbones are converging on the Transformer

The diffusion backbone started as the CNN-based UNet, but the newest generation (SD3, SD3.5, Sora, FLUX) increasingly adopts the DiT (Diffusion Transformer): cut the image into a patch sequence and process it directly with a Transformer. Generative AI's backbone is converging on a single architecture — the Transformer. It's a technical footnote that LLMs and image generation arrived at the same place by different roads.

4. Conditional Generation: Making Generation Controllable

Unconditional diffusion can only "draw a random picture"; what's actually useful is conditional generation — given a prompt, a reference image, or structural constraints, produce content that matches intent. Three levels, shallow to deep:

1. Text Conditioning: CLIP and the Text Encoder

The most basic condition is text. CLIP (Contrastive Language-Image Pre-training) embeds images and text into one semantic space via contrastive learning, making text embeddings alignable with image content; SD uses CLIP's text encoder to turn the prompt into embeddings, injected into the denoising network through cross-attention. The trade-off between conditional and unconditional is governed by Classifier-Free Guidance (CFG):

ε_θ = ε_θ_uncond + guidance_scale · (ε_θ_cond − ε_θ_uncond)

The larger guidance_scale, the closer the output hugs the prompt — and the lower the diversity (typical values 7–8). For the full text-image alignment mechanism, see Multimodal Models.

2. ControlNet: Structural Control

Text describes "what," but struggles with "where, and in what pose." ControlNet (2023) injects conditioning maps — edges, depth, skeletal pose, line art — directly into the denoising network:

 input image ─▶ edge/depth/pose extraction ─▶ conditioning map
                                                  │
                                     ControlNet (trainable copy of the backbone)
                                                  │
 prompt ─▶ text encoding ─▶ backbone UNet (frozen weights) ◀── conditioning features injected

ControlNet's idea: add a trainable branch alongside a frozen backbone, so conditioning signals plug into an already-powerful model at near-zero cost. The practical effect: given a person's skeletal pose, you can change outfit and background while keeping the pose — composition control became a designer's everyday tool.

3. LoRA: Low-Cost Customization of Styles and Characters

To generate "a specific artist's style" or "a specific character," you don't retrain the whole model. LoRA (Low-Rank Adaptation) trains only the low-rank matrices inserted into the backbone: a few dozen images and a single consumer GPU suffice for style customization, and at inference the few-MB weight files plug in and unplug at will:

Dimension	Full fine-tuning	LoRA adapter
Parameters to adjust	Billions	Millions (~1/1000)
Training VRAM	Multi-GPU A100	Single 12–24GB card
Training data needed	Tens of thousands of images	20–100 images
Composability	One per model	Multiple LoRAs stack and mix

LoRA has evolved from an LLM fine-tuning technique into the diffusion ecosystem's standard "style plugin." For fine-tuning methodology in depth, see Fine-Tuning and PEFT (LoRA); for the engineering side, read Fine-Tune Your Own LLM from Scratch.

5. Diffusion vs. GAN / VAE / Autoregression

Diffusion models didn't appear out of nowhere: they stand on the shoulders of VAEs and GANs, and divide labor with the autoregressive route. One table:

Dimension	GAN	VAE	Diffusion	Autoregressive
Core idea	Generator fools the discriminator	Compress–reconstruct	Noise–denoise	Predict the next token/patch one at a time
Generation quality	Good, but collapses easily	Tends toward blurry	Best today	Best for text, slow for images
Diversity	Poor (mode collapse)	Good	Good	Good
Training stability	Poor (adversarial instability)	Stable	Stable	Stable
Controllability (conditioning)	Medium	Medium	Strong (text/structure/graphs)	Strong (instruction-style)
Sampling speed	Fast	Fast	Slow (needs acceleration)	Medium (token by token, slow)

The one-line verdict: for high-fidelity image/audio/video generation, diffusion is the de facto standard; for flexible text generation, autoregression is king; VAE is the theoretical cornerstone and the latent-space infrastructure; GAN still matters in real-time small models and some scientific tasks. For where each route sits in the technical timeline, see A Brief History.

6. The Application Landscape: From Text-to-Image to World Simulators

Diffusion has penetrated nearly every content modality:

• Text-to-image: Midjourney, Stable Diffusion, DALL·E 3, FLUX, and more. Midjourney leads on aesthetic style, SD on its open-source ecosystem — see Midjourney and Image Generation;
• Text-to-video: Sora, Runway Gen-3, Kling, Veo, Luma, and more. Video is a "space + time" double task, demanding an order of magnitude more consistency and compute. Sora runs diffusion over video with spatiotemporal patches + DiT, and its technical report outright calls video generation models "world simulators" — see Sora and Video Generation;
• Audio generation: text-to-music (Stable Audio, AudioLDM), diffusion in TTS (DiffWave and friends), sound effects and voice cloning;
• 3D generation: DreamFusion, Point-E, TripoSR, TRELLIS, and more — from "text-to-3D model" to "image-to-3D asset," rewriting game and film asset pipelines;
• Image editing and restoration: img2img, inpainting, super-resolution — all rely on DDIM's deterministic sampling and conditioning injection;
• Scientific generation: molecular conformations, protein structures, materials design — diffusion's strength in continuous-space modeling is steadily migrating into science.

Modality	Representative systems	Backbone	Maturity
Image	Midjourney / SD / DALL·E 3	UNet → DiT	Commercially mature
Video	Sora / Veo / Kling	DiT + spatiotemporal patches	Catching up fast
Audio	Stable Audio / AudioLDM	UNet/DiT	Early commercial
3D	DreamFusion / TRELLIS	3D representation + diffusion	Research → production

7. Limitations and Controversies

Diffusion models are no silver bullet; five problems are unavoidable:

1. Compute cost: training iterates thousands of steps in latent space, inference still needs dozens of denoising steps, and a single video-generation inference costs far more than a text model. That's why "few-step sampling + distillation" (LCM, SDXL Turbo, ADD) became an industry-wide sprint;
2. Copyright and data compliance: Stable Diffusion trained on LAION-5B — a massive web-scraped image-text dataset full of copyrighted images — triggering landmark cases like Getty Images v. Stability AI and a class action by artists. The core dispute remains unresolved: does training itself infringe? Are outputs "derivative works"? The industry's answer is the "licensed data" route (Adobe Firefly trains only on licensed images), while at the mechanism level "what the model actually memorized" is still an open question. For the compliance framework, see AI Safety and Governance;
3. Recurring detail failures like "hands/text rendering": early SD models regularly botched fingers and text. Causes include: latent-space compression discards high-frequency detail, UNet attention under-models small local structures, and training data rarely covers certain poses and fonts. Improvements came from bigger, stronger models (SDXL's hands clearly improved), stronger VAEs, DiT's global modeling, and targeted data cleaning;
4. Evaluation is hard: generation has no "ground truth." FID (Fréchet Inception Distance) measures the feature distance between generated and real distributions but can be gamed by "memorizing training images"; CLIP Score measures text-image alignment but says nothing about aesthetics; in the end you still need human side-by-side (double-blind) comparison. For the evaluation framework, see Evaluation and Validation;
5. Deepfakes and the authenticity crisis: face swaps and fabricated videos break "seeing is believing," with C2PA, SynthID, and other watermarking schemes locked in a running battle against detection techniques. Practitioners must add content filtering and labeling standards at the deployment layer.

Generation ≠ Truth

A diffusion model's optimization target is "distributional similarity," not "factual correctness." AI-generated images/video can be creative tools — or tools for misinformation and infringement. Always position them as "creative artifacts that require verification and governance."

8. Diffusion and LLMs: Pixels to Diffusion, Language to the LLM

A common mistake is equating "generative AI" with "large language models." In mainstream multimodal systems the two have a clear division of labor:

• Diffusion models handle "generating pixels": translating semantic conditions into high-fidelity images, video, and audio;
• The LLM handles "language understanding and instruction": parsing user intent, rewriting/optimizing prompts, planning generation tasks — the "scheduling brain";
• The text encoder is the bridge: CLIP, T5, and friends convert language-side representations into condition vectors the diffusion side can inject.

A few real collaboration cases:

• Sora: used DALL·E 3-style recaptioning to "translate" video data into high-quality text annotations, then ran video diffusion conditioned on text;
• Stable Diffusion 3 / FLUX: switched to stronger text encoders like T5, markedly improving adherence to complex prompts (multiple objects, multiple relations);
• Agentic workflows: the LLM plans "what to draw first, in what style, how many iterations"; the diffusion model executes — the two combine through tool calls into a full AI agent system.

Architectural convergence

On the image side, backbones moved from UNet to DiT; on the text side, it was Transformer all along. In training paradigms, diffusion's "predict the noise" and the LLM's "predict the next token" both amount to "recover structure from perturbation." The two are converging on the same Transformer foundation — understand Transformers and Attention and LLMs, and you've grasped the shared substrate of the multimodal generation era.

9. Trade-offs and Choices

Practitioners constantly navigate these four tensions:

• Speed vs. quality: sampling steps, model size, and distillation form a triangle. LCM/Turbo buy real-time performance at the cost of detail and diversity; "fast and good" only comes from more compute;
• Controllability vs. diversity: stronger CFG guidance and tighter ControlNet constraints push outputs closer to intent but more homogeneous. Art wants diversity; product launches want consistency — tune the parameters per scenario;
• Open-source self-hosting vs. closed API: the SD family is controllable, customizable, and free of usage billing, but you run the GPUs, VRAM, and safety filtering yourself; Midjourney and DALL·E 3 work out of the box but offer no low-level control. This is a cost-structure decision, not a technology preference;
• Base model vs. fine-tuned ecosystem: a general base covers breadth but excels at nothing specific; community fine-tunes (DreamShaper, photorealism-oriented models) beat it on certain content while narrowing generalization. The common production setup is "one base + a handful of LoRAs."

Further Reading

• Multimodal Models — unified representations and conditional generation across text, images, and video
• Fine-Tuning and PEFT (LoRA) — LoRA style-customization methodology in the diffusion ecosystem
• Large Language Models — the half that handles "language understanding and instructions"
• Transformers and Attention — the architectural source of cross-attention and DiT
• Midjourney and Image Generation — the flagship text-to-image case study
• Sora and Video Generation — text-to-video and world simulators
• Evaluation and Validation — a complete framework for evaluating generation quality
• AI Safety and Governance — copyright, deepfakes, and compliance governance
• AI Agents — LLM planning + diffusion generation as a combined system
• A Brief History — the diffusion timeline from statistical physics to text-to-video
• Model & Leaderboard Cheat Sheet, Glossary — track new models, look up terms
• Core Paper Reviews — paper-by-paper breakdowns of DDPM, LDM, DiT, and more

References

• Sohl-Dickstein et al. Deep Unsupervised Learning using Nonequilibrium Thermodynamics (ICML 2015) — the origin of the diffusion idea
• Ho, Jain, Abbeel. Denoising Diffusion Probabilistic Models (DDPM, NeurIPS 2020) — the landmark paper that made diffusion a practical generative tool
• Song, Meng, Ermon. Denoising Diffusion Implicit Models (DDIM, ICLR 2021) — deterministic accelerated sampling
• Song et al. Score-Based Generative Modeling through Stochastic Differential Equations (ICLR 2021) — the SDE unified framework for diffusion/score models
• Rombach et al. High-Resolution Image Synthesis with Latent Diffusion Models (LDM, CVPR 2022) — Latent Diffusion, Stable Diffusion's direct ancestor
• Radford et al. Learning Transferable Visual Models From Natural Language Supervision (CLIP, 2021) — the text encoder behind image-text alignment
• Zhang, Rao, Agrawala. Adding Conditional Control to Text-to-Image Diffusion Models (ControlNet, ICCV 2023) — structural conditioning injection
• Hu et al. LoRA: Low-Rank Adaptation of Large Language Models (ICLR 2022) — the original LoRA paper, later widely adopted by the diffusion ecosystem
• Peebles, Xie. Scalable Diffusion Models with Transformers (DiT, ICCV 2023) — the Transformerization of diffusion models
• Brooks et al. Video generation models as world simulators (OpenAI technical report, 2024) — the Sora technical report
• Lilian Weng. What are Diffusion Models? (blog) — an intermediate-level survey with complete derivations
• Jay Alammar. The Illustrated Stable Diffusion (blog) — a visual walkthrough of the SD architecture
• Hugging Face. Diffusers official documentation — a one-stop manual for samplers, pipelines, and training scripts