ML Handbook

Theme

Diffusion Models and Generative AI

Quick overview From DDPM to Stable Diffusion: this article dissects the mathematical principles of diffusion model denoising/counter-denoising, latent space architecture, and sampling acceleration methods, clarifies conditional control techniques like ControlNet and LoRA, and provides local practice code and risk assessment.

Diffusion Models and Generative AI ​

In August 2022, Stable Diffusion put the ability to "input a sentence, generate an image" into everyone's hands in an open-source manner. Just a sentence — "a fox sitting in a snowfield, golden hour" — and anyone — no need to know how to draw, nor even to understand machine learning — can get an almost-indistinguishable-from-reality image within dozens of seconds. Behind it is a type of generative model called a diffusion model, which first destroys data, then learns to rebuild it, turning "generation" into "gradually reconstructing from noise."

This article follows the path of "why needed → how it works → how to accelerate → how to control → how to evaluate → what risks → how to get started" to thoroughly explain diffusion models. It is the most successful of the three generative model routes (VAE/GAN/diffusion), and is naturally another victory of deep learning and Transformer in the image domain.

1. Generative AI Landscape: From "Can Paint" to "Can Work" ​

The 2020–2024 timeline of generative AI is a dizzying one. In just four years, image and video generation went from "lab curiosities" to "productivity tools":

TimeEventSignificance
2021.01OpenAI releases DALL·E (12B parameters)Text-to-image enters the public eye, but closed-source, requires queuing
2022.04OpenAI releases DALL·E 2Qualitative leap in realism, but still closed-source
2022.07Stability AI releases Stable Diffusion 1.4First open-source high-quality text-to-image; ecosystem explodes
2022.08Midjourney V3/V4 go liveAesthetic style peaks; community works retroactively define "AI art"
2023.07SDXL, SDXL Turbo released1024×1024 HD + one-step generation
2023.11ControlNet, LCM, and other control/acceleration techniques maturePrecise control over composition; generation shifts from "lottery" to "painting"
2024.02OpenAI releases Sora demoMinute-level realistic video; text-to-video reaches an inflection point
2024 to presentRunway Gen-3, Kling, Veo, SVD, etc.Video generation enters the usable stage; competition heats up

Several trends worth noting:

First, has text-to-image "peaked"? The gap in "single-image quality" between open-source (Stable Diffusion series) and closed-source (Midjourney, DALL·E 3, Imagen) is rapidly narrowing; the competitive focus shifts to controllability (ControlNet, region editing), speed (one-step generation), and workflow integration (Photoshop, Blender, game asset pipelines).

Second, video generation is the next battlefield. Image is a "spatial" task; video is "space + time," demanding orders-of-magnitude higher consistency and compute. Sora's technical report directly calls video generation models "world simulators" — capable of learning physical laws and causal relationships, which goes beyond the scope of "generation," see Section 6 of this article.

Third, generative AI is reshaping industry roles. Designers shift from "hand-drawing each image" to "prompting + screening + refining"; game companies use generative models to batch-produce concept art and textures; e-commerce uses image-to-image to generate product showcases. The infrastructure behind this transformation is the protagonist of this article — diffusion models. For the complete narrative of "why machine learning reached today," read What is Machine Learning first.

2. Diffusion Model Principles: "Destroying" and Playing in Reverse ​

1. Core idea ​

The idea of diffusion models (Diffusion Probabilistic Models) comes from statistical physics, first proposed by Sohl-Dickstein et al. (2015). In 2020, Ho et al.'s DDPM (Denoising Diffusion Probabilistic Models) made it a practical tool that crushed GANs.

Its idea is very simple, explainable with one metaphor:

"Destroying" a photo (forward process, completed during training, extremely cheap)
  Real image x₀ ──add noise──▶ x₁ ──add noise──▶ … ──add noise──▶ x_T ≈ pure Gaussian noise

"Playing destruction in reverse" (reverse process, what the model must learn)
  Pure noise x_T ──denoise──▶ x_{T-1} ──denoise──▶ … ──denoise──▶ brand-new image x₀

The key insight is: adding Gaussian noise to an image step by step is a "cheap" and "reversible" destruction process. Cheap — because x_t at any time t can be directly sampled in closed form from x₀, without iterating step by step; reversible — although the inverse transform of a single denoising step is unknown, you can train a neural network to predict "what noise was added at this step," thereby learning to play destruction in reverse.

Thus "generation" is transformed into "denoising": start from pure noise, let the network gradually remove noise step by step, and ultimately produce a brand-new image that conforms to the training data distribution P(x). This is much more stable than GAN's adversarial game and simpler than VAE's variational inference — this is the fundamental reason diffusion models came out on top. For a detailed comparison of the three routes, see Generative Models.

2. Forward noising: with closed-form solution ​

Let x₀ be a real image; a noise schedule gives the variance β₁, β₂, …, β_T for each step (typically β increases linearly from 1e-4 to 0.02). The forward process is defined as:

q(x_t | x_{t-1}) = N( x_t ; √(1-β_t)·x_{t-1}, β_t·I )

Because of Gaussian additivity, the state after t steps doesn't require step-by-step iteration — it can be written directly:

x_t = √(ᾱ_t)·x₀ + √(1-ᾱ_t)·ε,   ε ~ N(0, I)

where α_t = 1 − β_t, ᾱ_t = ∏_{s=1}^{t} α_s. Here √(ᾱ_t) controls "how much of the original image remains," and √(1−ᾱ_t) controls "how much noise has been mixed in." When T is large enough (DDPM uses T=1000), ᾱ_T → 0, and x_T converges to standard Gaussian noise N(0,I).

x_t = √(ᾱ_t) · x₀ + √(1-ᾱ_t) · ε
      └── original image component ──┘   └──── noise component ────┘
      t=0: pure original image           t=T: pure noise (ᾱ→0)

During training, for each sample, randomly draw a time t and use this formula to instantly get the noised image x_t and "the noise added at this step" ε — this is why diffusion model training is so cheap.

3. Reverse denoising: DDPM's training objective ​

We train a neural network ε_θ (usually a UNet, see Section 3) to predict the noise ε mixed into x_t. DDPM's simplified training objective (a tractable variant of the lower bound on the true log-likelihood) is:

L = E_{t, x₀, ε} [ ‖ ε − ε_θ( √(ᾱ_t)·x₀ + √(1-ᾱ_t)·ε , t ) ‖² ]

Translated into plain language: for each training sample, randomly pick a time t, add noise, then have the network predict "the mixed-in noise" from "the noised image + time t," measuring prediction accuracy with MSE. The network also receives time t as input (timestep embedding) because "the intensity of denoising" varies with time — the larger t, the more noise, the bolder the denoising.

Why "predict noise" instead of "predict the image"?

They are mathematically equivalent: given x_t and ε, you can back-calculate x₀. But experiments show that predicting noise gives more stable training signals with smaller variance. This is one of the key engineering improvements of DDPM over earlier diffusion models (which directly learned the data gradient/score). From the "score matching" perspective, what ε_θ learns is precisely the gradient of the data's log density ∇_x log p(x) — see Song et al.'s unified SDE framework (arXiv:2011.13456).

4. Sampling: turning noise back into images ​

After ε_θ is trained, generation starts from x_T ~ N(0,I), repeatedly executing the reverse step, for T steps total:

x_{t-1} = (1/√α_t)·( x_t − (β_t/√(1-ᾱ_t))·ε_θ(x_t, t) ) + σ_t·z,  z ~ N(0,I)
          └──── remove the predicted noise, restore a "cleaner" image ─────┘   └─random perturbation─┘

where σ_t is the random jitter at each step (ensuring the process is a true diffusion reverse). Each step "removes a bit of noise." Observing visually, the first dozens of steps show contours emerging from chaotic noise, and the last dozens refine "texture" — this is why the number of sampling steps (num_inference_steps) directly determines generation quality and cost.

Sampling process (during generation, T≈1000 steps can be accelerated to 20–50 steps):
Pure noise ─▶ blurry blocks ─▶ contours appear ─▶ structure forms ─▶ details sharpen ─▶ final image
  x_T                                              x_1      x₀

At this point, you've mastered the entire mathematical skeleton of diffusion models: forward closed-form noising + reverse learning denoising + iterative sampling. Below we see how it's engineered into a product that can generate images.

3. Latent Space and Stable Diffusion Architecture: VAE + UNet + CLIP ​

DDPM has a fatal flaw: it's slow. Pixel-level diffusion iterates 1000 steps in 512×512×3 image space, making training and inference too expensive for commercial use. Latent Diffusion Models (LDM, Rombach et al., 2022) solve this with one move: don't diffuse in pixel space, diffuse in a compressed latent space. This is the direct predecessor of Stable Diffusion.

1. Why latent space ​

Images are highly redundant: adjacent pixels are highly correlated, and "semantic information" is far fewer than pixels. First use an autoencoder (VAE) to compress the image into a low-dimensional latent space, then denoise — compute drops by an order of magnitude:

Pixel-space diffusionLatent-space diffusion (LDM/SD)
Operating dimension512×512×3 ≈ 780K dims64×64×4 ≈ 16K dims
Relative compute~1About 1/50
Generation resolutionHard to break 128×128Easily 512×512 to 1024×1024

Even better, the VAE's latent space is a semantically meaningful continuous space: interpolating, editing, or mixing in latent space corresponds to smooth, natural transformations in images — this provides the stage for all subsequent conditional control techniques (Section 5).

2. The three components' division of labor ​

Stable Diffusion (SD 1.x) consists of three modules, each with its own role:

                     ┌─────────────────────────────────────────┐
   Text prompt ──▶ CLIP text encoder ──▶ text embedding (77 tokens × 768 dims)
    "a red fox..."                                            │
                                                                ▼
   Random noise z_T ──▶ UNet (latent-space denoising)──▶ latent z₀ ──▶ VAE decoder ──▶ image
                         ▲  ▲                               │
                         │  └── cross-attention injects text condition  │
                         └────── timestep embedding (which step)       ┘
ComponentRoleNotes
VAE encoderImage → latent space (8× compression)Used for generating training samples during training
VAE decoderLatent → pixel imageFinal step during generation
UNetStepwise denoising in latent space (predict noise ε_θ)The model's "brain"; SD 1.5 has about 860M parameters
CLIP text encoderPrompts → text embeddingsSD 1.x uses CLIP ViT-L/14
Cross-Attention"Feed" text conditions into UNet layersThe core mechanism of text-to-image capability

UNet is an encoder-decoder convolutional network with skip connections — the encoder progressively downsamples to extract multi-scale features, the decoder progressively upsamples to restore, and skip connections preserve details. Diffusion models adopted its architecture, long proven in medical image segmentation, then added time embedding and text cross-attention at each layer.

Cross-Attention is the magic of text-to-image: the UNet's intermediate features serve as queries (Query), text embeddings serve as keys/values (Key/Value), and through attention mechanisms, "each position in the image goes to look at the words relevant to it." This is precisely the multi-head attention idea from Transformers transplanted to image generation — images and text are first deeply coupled at the feature level.

Relationship with the Transformer family

The backbone of diffusion models was initially CNN's UNet, but the latest generation (SD3, Sora, SD 3.5, etc.) increasingly adopts DiT (Diffusion Transformer): cut images into patches and process directly with Transformers. The skeleton of generative AI is converging to the same architecture — Transformer. This is also a technical footnote to how large language models and image generation converge from different paths.

3. Training data and model zoo ​

SD 1.x was trained on a subset of LAION-5B (5.85 billion image-text pairs, Schuhmann et al., 2022), with weights open-sourced under the CreativeML OpenRAIL-M license. The community built a vast ecosystem around SD:

  • SD 1.5: the classic mainstay, 512×512, richest ecosystem (LoRA, ControlNet all support it first);
  • SD 2.1: switched to OpenCLIP text encoder, but community adoption was actually lower than 1.5;
  • SDXL: 1024×1024, dual text encoders (OpenCLIP ViT-bigG + CLIP ViT-L), UNet with about 2.6B parameters, significantly improved style and details;
  • SD3 / SD3.5: switched to DiT backbone + three text encoders (including T5), introduced rectified flow training;
  • Companion fine-tuned models: DreamShaper, Realistic Vision, etc., all specialty models trained by the community on base models.

Methods to obtain these models and datasets can be found in Datasets and Tools.

4. Sampling Acceleration: Turning "Slow" into "Fast" ​

DDPM sampling needs 1000 steps, generating an image takes dozens of seconds to minutes. The entire acceleration spectrum revolves around one goal: get the same quality with fewer denoising steps.

1. DDIM: a smarter sampling trajectory ​

DDIM (Denoising Diffusion Implicit Models, Song et al., 2021) discovered that DDPM's reverse sampling trajectory isn't unique — the forward process defines a "family of distributions from x₀ to x_T," while the reverse process can have infinitely many. DDIM constructed a non-Markovian reverse process and gave a key result:

DDIM reverse step:
x_{t-1} = √ᾱ_{t-1}·( x_t − √(1−ᾱ_t)·ε_θ )/√ᾱ_t + √(1−ᾱ_{t-1}−σ_t²)·ε_θ + σ_t·z
  • When the random term coefficient σ_t = 0, sampling is deterministic: the same noise input always produces the same image — this gave rise to practical features like "image-to-image" and "inpainting";
  • Large-step jumping: sample only at 20–50 selected time steps, with small quality loss.

DDIM reduced text-to-image from "one image per minute" to "one image per few seconds," the first order-of-magnitude acceleration.

2. Distillation: teaching the few-step model from the teacher model ​

Distillation's idea is to train a "student" model that mimics the "teacher" (1000-step model)'s sampling behavior:

  • Progressive Distillation: halve the steps round by round, with each round's student learning from two teacher outputs;
  • ADD (Adversarial Diffusion Distillation, 2023): distillation + adversarial loss, making 1–4 step outputs approach the teacher's quality — SDXL Turbo used this to achieve near-SDXL quality in one-step generation;
  • Consistency Distillation / LCM (Latent Consistency Models, 2023): directly learn a consistent mapping "from any time x_t to one step jumping to x₀," paired with a special sampler (e.g., LCMScheduler), producing images in 4–8 steps.

3. Mathematical accelerators: DPM-Solver, etc. ​

There's also a class of no-retraining-needed methods: treating the denoising process as a numerical solution problem for an ODE (probability flow ODE), using high-order numerical methods to reduce truncation error. DPM-Solver (2022), UniPC, and other samplers approach 1000-step DDPM quality in 10–20 steps and are regulars of diffusers' default samplers.

MethodStepsRequires retraining?Notes
DDPM1000NoBaseline, slow
DDIM20–50NoDeterministic, editable, most universal
DPM-Solver / UniPC10–20NoMathematical acceleration, plug-and-play
LCM1–8Yes (distill small models)Real-time generation, suitable for interactive scenarios
ADD / SDXL Turbo1–4YesOne-step generation, best quality

The cost of acceleration

Few-step sampling doesn't lose much in "overall image structure," but significantly degrades in high-frequency details (text, small objects, fingers); distilled models often also sacrifice diversity (images from the same prompt look more similar). "Fast" and "good" represent a real tradeoff — there's no free lunch.

5. Conditional Control: Making the Model "Obey" ​

Basic text-to-image has an experience pain point: uncontrollable. The same prompt, composition, pose, and lighting all depend on luck, jokingly called "pulling lottery tickets" by the community. Conditional control techniques turn "pulling lottery" into "painting."

1. ControlNet: precise structural control ​

ControlNet (Zhang et al., 2023)'s goal is: additionally provide a condition image (edges, depth, pose, segmentation map), making the generated image follow its structure.

The approach is elegant: copy a trainable version of the UNet's encoder, injecting condition image features into the original UNet layer by layer through zero convolution (1×1 convolutions with initially zero weights). Zero convolution guarantees the copy outputs 0 at the start of training, not destroying any capability of the pre-trained model; the condition signal gradually "takes over" structure generation as training progresses.

User input: sketch line drawing / Canny edge / depth map / pose skeleton / semantic segmentation map
                      │
                      ▼
         Trainable ControlNet copy ──zero conv──▶ inject into original UNet at each layer
                      ▲
             Original UNet (frozen) ◀──── text prompt

Typical applications: designers sketch a line drawing and let AI color-render, fix indoor layouts with depth maps, control human poses with pose skeletons, arrange scenes with semantic segmentation maps. ControlNet was the most impactful technology for text-to-image workflows in 2023.

2. LoRA: customizing style on a budget ​

To "teach the model to depict a certain character or art style," the traditional approach is fine-tuning all weights — expensive VRAM, prone to catastrophic forgetting. LoRA (Low-Rank Adaptation) offers another answer: freeze the original model, only add a low-rank bypass around the attention layer weights.

W' = W + ΔW = W + B·A,   B ∈ R^{d×r}, A ∈ R^{r×d}, r usually 4~64
     └─original weights (frozen)─┘   └────────low-rank increment (only this is trainable)────────┘

LoRA itself originated from large language model fine-tuning (Hu et al., 2021) and was transplanted wholesale to diffusion models: training VRAM demand drops from "several GB for full fine-tuning" to "a few hundred MB," training data drops from "hundreds of thousands of images" to "a few dozen example images," and the output is a 50–500MB .safetensors file, dynamically loaded, stacked, swapped during sampling — Civitai's ecosystem of hundreds of thousands of LoRA models is born of this mechanism.

3. IP-Adapter: letting "reference images" speak ​

LoRA and ControlNet control "style/structure," while IP-Adapter (2023) solves "generate referencing an image": pass the reference image through the CLIP image encoder to get an image embedding, which is injected into the UNet in parallel with the text condition via a decoupled cross-attention branch. The result is the ability to use a reference image to specify "what the protagonist looks like," while using text to specify scene and action — "image-to-image" upgrades from simple style transfer to "identity-preserving" level control.

TechniqueControl dimensionTypical scenarioTraining effort
Prompt / CFGSemanticsDaily generation0
ControlNetGeometry/structureLine drawings, poses, depthMedium (needs paired data)
LoRAStyle/character/objectStyle customizationSmall (a few dozen images)
IP-AdapterReference image identityKeep character consistentMedium

Combining them creates productivity

Real workflows are almost always combo punches: ControlNet fixes structure + LoRA fixes style + IP-Adapter maintains character consistency + prompts define semantics, finally using image-to-image (img2img) or inpainting to refine locally. This entire "model factory" building approach is also a classic topic worth writing into Portfolio Projects.

6. Text-to-Video and Multi-Modal Trends: From Static to World Simulators ​

Video generation is diffusion models' "time dimension extension" and the most fiercely contested frontline of current generative AI.

1. Two additive approaches for the technical routes ​

Turning image diffusion into video diffusion, the core question is only one: how to model the time dimension. Two mainstream routes:

  • 3D convolution / spatiotemporal attention: extend 2D convolutions and attention with a time dimension, directly learning the joint distribution of "video frame sequences." Stable Video Diffusion and early Runway versions go this route;
  • Insert time layers into a frozen text-to-image model: AnimateDiff's approach — keep the mature text-to-image model unchanged, only inserting trainable temporal attention modules between layers. The benefit is reusing image priors with smaller training data needs.

2. Sora and DiT: video as "spatiotemporal patches" ​

OpenAI's Sora, released in February 2024, was an inflection point not just for video length (minute-level) and realism, but for architecture and paradigm upgrade:

  • Cut video into spatiotemporal patches: the same idea as ViT cutting images into patches and LLMs cutting text into tokens — Transformers treat "any length of token sequences" equally, so images, videos, audio, and text can for the first time be placed into the same generative framework;
  • DiT backbone: the denoising network switched from UNet to Diffusion Transformer (Peebles & Xie, 2022), supporting arbitrary resolution/duration;
  • re-captioning: use DALL·E 3's method to give training videos detailed text re-labeling, greatly improving "understanding prompts" capability;
  • The technical report calls such models "world simulators": they can spontaneously develop rough "physical intuition" for gravity, lighting, and object interactions.

3. Open problems in video generation ​

Even by 2025, video generation still faces three hard constraints: consistency (characters, scenes, props don't drift across frames), square-level computational cost from duration, and causal correctness (e.g., "objects should maintain their state when occluded and reappear" — such long-range memory). Its maturity is far behind text-to-image — but this precisely means greater research space and engineering opportunities.

Another major multi-modal trend is unified generation: Google's Gemini and OpenAI's GPT-4o series put "understanding" and "generation" into the same model, with images, videos, audio, and text able to convert and edit each other. The backbone of this direction is still the victory of deep learning and Transformers.

7. How to Evaluate "Is the Generation Good?" ​

Evaluating generation quality is much harder than classification accuracy — because "looks good" is subjective, and there's no standard answer. In practice, two classes of metrics are used: automatic metrics + human evaluation.

1. FID: the most common automatic metric ​

FID (Fréchet Inception Distance) works by: using a pre-trained Inception-V3 to map both real and generated images into feature space, then computing the Fréchet distance between the two feature distributions (approximated as Gaussians):

FID = ‖μ_r − μ_g‖² + Tr( Σ_r + Σ_g − 2·(Σ_r·Σ_g)^{1/2} )
      └─distance between real/generated feature means─┘   └─────difference in feature covariances──────┘

Lower FID means the two image feature distributions are closer. It penalizes both "poor quality" (mean shift) and "lack of diversity" (covariance narrowing), making it the standard metric for text-to-image papers.

But FID has well-known defects:

  • Can be gamed with "memorization": if the model directly "memorizes" images from the training set, FID is still very low — FID can't distinguish "generation" from "copying";
  • Insensitive to mode collapse: generating only "one type of nice-looking image" can still yield a beautiful FID;
  • Feature-space bias: Inception-V3 features lean toward object recognition, so style-level differences may be underestimated.

Supporting metrics include: IS (Inception Score) evaluates single-image sharpness and category diversity (but is unrelated to the real distribution); CLIP Score measures "image-text alignment" (how well the generated image matches the prompt). A complete framework for metric choice is in Model Evaluation and Validation.

2. Human evaluation: the "human judge" you can't bypass ​

Automatic metrics never replace human eyes, especially when involving aesthetics, style, and whether it matches intent:

  • A/B testing and head-to-head matchups: give reviewers two images and ask them to pick the better one (Midjourney's community tournament is a scaled-up version of this mechanism);
  • Elo ranking: run an Elo system on pairwise comparisons (similar to chess ratings), obtaining a relative strength ranking for models — LMSYS's Chatbot Arena validated this method on the text side;
  • Task-oriented evaluation: e.g., "fix an image / complete an in-painting task," let professional users score.

An honest reminder on evaluation

Any single metric can be gamed. The rigorous approach is: FID/CLIP Score for quick screening, final decisions based on double-blind human comparisons with hundreds of participants. Also don't forget to evaluate engineering metrics like generation speed, VRAM usage, and failure rate — a model with a lower FID but 10× slower isn't necessarily better in production. This is exactly what Section Evaluation and Validation repeatedly emphasizes: "metrics must serve objectives."

8. Risks and Ethics: The Other Side of Technology ​

The stronger diffusion models' power, the higher the cost of misuse. This section clarifies three unavoidable issues, plus industry's response paths.

1. Deepfakes and the authenticity crisis ​

Face-swapping, faked celebrity statements, synthesized fake news — diffusion models made "seeing is believing" no longer valid. Starting from 2023, events like "AI-generated photo of the Pope in a puffer jacket" and "AI-synthesized photo of Trump being arrested" went viral time and again. Industry responses fall into two layers:

  • Passive detection: train classifiers to distinguish real from fake, but generative tech's progress keeps detection perpetually behind (adversarial generation);
  • Active labeling / watermarking: C2PA Content Credentials (led by Adobe, joined by OpenAI, Microsoft, etc.) records an "AI-generated" provenance chain in file metadata; SynthID (Google DeepMind) embeds invisible watermarks into pixels, detectable even after cropping/compression. Starting from 2023, major model vendors have progressively forced labeling of generated content.

2. Copyright: gray areas in training data ​

Stable Diffusion was trained on LAION-5B — a massive image-text pair dataset scraped from the internet, containing many copyright-protected images. This triggered two landmark lawsuits:

  • Getty Images v. Stability AI (2023): Getty alleged its model output images with Getty watermark traces, constituting direct infringement;
  • Artist class action (Andersen et al. vs. Stability AI, etc.): three artists alleged the model "copied art styles."

The core legal controversy remains largely unresolved: does training constitute infringement? Is the model's output a "derivative work"? The practical industry response is the "licensed data" route (Adobe Firefly trains only on Adobe Stock-licensed images; OpenAI partners with Shutterstock), but the larger debate — the relationship between generated content and human creation — is far from over. Mechanistically, what the model "memorized" vs. "learned" is a question the interpretability field is still answering.

3. Content safety and identity misuse ​

  • NSFW and violent content: SD weights come with an optional safety checker (an NSFW classifier), but once the open-source model is republished, this defense line is effectively nullified;
  • Portrait rights and identity theft: with LoRA, anyone's face can be replicated from a few dozen photos — "deepfake pornography" and "AI fraud videos" have become real social cases;
  • The dilemma of moderation: over-moderation suppresses legitimate creation (medical, artistic, educational scenarios); under-moderation fuels misuse. This is a "governance dilemma" unique to generative AI.

Specific actions practitioners can take: watermark/attach metadata to generated images, add configurable input-output filtering at the deployment layer, implement real-name and authorization verification for "face generation" features, and establish AIGC content labeling norms within the organization. Technology itself is neutral; tradeoffs are engineering and human decisions.

9. Practice: Running Stable Diffusion Locally ​

Enough theory — let's code. Below we use Hugging Face's diffusers library to run Stable Diffusion locally — one command to install, one script to generate.

1. Environment setup ​

bash
pip install diffusers transformers accelerate safetensors torch
# If you have an NVIDIA GPU (≥ 8GB VRAM recommended, 16GB+ preferred), install the CUDA version of torch:
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121

Model weights will auto-download on first run (SD 1.5 ~4GB, SDXL ~7GB) to ~/.cache/huggingface. Most SD weights use gated licenses; you need to log in to Hugging Face first and accept the model card terms:

bash
huggingface-cli login   # paste your HF token

2. Minimal text-to-image script ​

python
import torch
from diffusers import StableDiffusionPipeline

# Load the model. Model repos are often gated; you must first accept the license and log in.
# Alternative: stabilityai/stable-diffusion-xl-base-1.0 (1024×1024, more VRAM-hungry)
pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,   # Half precision, saves half the VRAM
    safety_checker=None,          # For demo only; production should keep it
)
pipe = pipe.to("cuda")            # Replace with "cpu" if no GPU, but 10–50× slower

prompt = "a red fox sitting in a snowfield, golden hour, photorealistic, high detail"
image = pipe(
    prompt,
    num_inference_steps=30,       # Sampling steps: the knob between quality and speed
    guidance_scale=7.5,           # CFG guidance strength: higher = closer to prompt, less diversity
).images[0]

image.save("fox.png")

This code is the complete landing of the architecture from Section 3: from_pretrained pulls the full VAE+UNet+CLIP set, and pipe(prompt) internally completes "text encoding → noise sampling → UNet stepwise denoising → VAE decoding" in sequence.

3. Image-to-image / switching samplers ​

python
from PIL import Image
from diffusers import StableDiffusionImg2ImgPipeline, DDIMScheduler

pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16
).to("cuda")

# Switch to a faster deterministic sampler (corresponds to DDIM in Section 4)
pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)

init = Image.open("sketch.png").convert("RGB").resize((512, 512))
out = pipe(prompt="watercolor painting of a mountain lake",
           image=init,
           strength=0.6,          # 0~1: how much structure from the original image to retain
           guidance_scale=7.5).images[0]
out.save("painting.png")

4. Customizing style with LoRA ​

python
# Load a LoRA adapter onto the already-loaded pipe (HF repo or local .safetensors)
pipe.load_lora_weights("ostris/super-cute-cat-playground")  # example repo
# Multiple LoRAs can be stacked; use pipe.unload_lora_weights() to unload back to the original model

If you want to train LoRA yourself, diffusers' official repo includes training scripts (train_dreambooth_lora.py, etc.); with 20–100 images of your target style, training can be done on a single 12–24GB VRAM GPU — see the diffusers training docs.

5. Common troubleshooting quick reference ​

SymptomCause and fix
CUDA out of memorySwitch to torch_dtype=float16, lower resolution, reduce num_inference_steps, use enable_attention_slicing()
Output is pure noiseWeights not fully downloaded or model mismatches pipeline; delete cache and re-download
Output is gray / has grid artifactsSwitch to safetensors weights, convert pipe.vae back to float32 before sampling
SlowClose other VRAM-consuming programs, upgrade GPU; or switch to LCM/Turbo fewer-step models
Prompt not taking effectConfirm the text encoder loaded normally; CLIP has a 77-token limit, and overly long prompts are truncated

Next steps

After running the minimal example, build a small project around "ControlNet line-drawing colorization" or "SDXL + LoRA character consistency," paired with effect comparisons and evaluation (FID + human blind test) — that's a great ML portfolio project. If any terms feel unfamiliar, check the Glossary anytime.

10. Tradeoffs and Decision Points ​

Four tensions that repeatedly appear in diffusion model projects, worth keeping in mind:

  • Speed vs. quality: steps, model scale, and distillation degree form a triangle — LCM/Turbo trade real-time for detail and diversity; to "be fast and good," only add money (larger distilled models, stronger GPUs).
  • Controllability vs. diversity: increasing CFG guidance strength (guidance_scale) and conditional control (ControlNet) makes outputs strictly match intent, but compresses the diversity of the generation distribution, making results converge and lose surprise. Artistic creation needs diversity; product deployment needs consistency — tune parameters by use case.
  • Base models vs. fine-tuned models: general base models (SDXL) have broad coverage but aren't specialty-optimized; specialty fine-tunes (DreamShaper, realism models) are clearly superior in certain content types but have narrower generalization. Production environments are typically "one base + several LoRAs," not one model for everything.
  • Open-source self-deployment vs. closed-source API: open-source (SD series) is controllable, customizable, no per-usage fees, but you manage GPU, VRAM, and safety filtering yourself; closed-source (Midjourney, DALL·E 3) is plug-and-play with stable quality, but lacks low-level control, and data cross-border and vendor dependence are concerns. For teams, this is a cost structure question, not a technical preference.

Two more commonly overlooked tradeoffs: training cost vs. inference cost (expensive once, cheap each inference, or vice versa — decides where your compute budget should go); capability vs. responsibility (the stronger the generation, the larger the engineering investment in moderation, watermarking, and compliance — this cost is often underestimated by new teams).

11. Further Reading ​

References ​