MODEL DEPLOYMENT HANDBOOK
The systematic knowledge you need to take a model from training complete to reliably serving traffic — inference engines · model compression · serving and deployment architecture · performance optimization · monitoring and MLOps · LLM inference · career paths
These 50+ pages aren't a library to read cover to cover — they're routes you can combine as needed
Interviewing within 1–3 months: go straight to JD benchmarking, resume rewrites, and the interview question bank. Every page maps to what top companies actually test.
View path →Time to build a solid foundation: work from the Guide through Practice on a weekly schedule, with clear checkpoints for each week.
View path →For working builders who won't read cover to cover: use the "problem → page" index to look things up and get back to work. Treat it as a reference book.
View path →If you only read ten pages, read these
What does a production-grade inference system actually consist of? This page dissects the seven-layer structure — request entry, serving layer, inference engine, resource layer, plus the cross-cutting concerns of data and versioning, observability, and governance — covering each layer's responsibilities, common failure points, and how the layers cooperate.
GuideModel deployment turns a trained model into a production system that keeps serving predictions. This article gives a precise definition, the four elements of deployment, the fundamental difference from training, the unique challenges of model systems (drift, quality degradation), and what a deployment engineer actually solves.
GuideInference is every forward pass a model makes once it's live. This article covers what inference really is, how it fundamentally differs from training, the latency breakdown of a single inference request, and what inference engines actually do—graph optimization, operator fusion, and kernel selection.
Core ConceptsLLM inference is the hottest frontier in model deployment: autoregressive generation, KV cache, continuous batching, speculative decoding, tensor parallelism. This article lays out the GPU memory ledger of LLM serving and every optimization technique that matters.
Core ConceptsA model deployment without monitoring is flying blind. This article covers the four golden signals for inference services, ML-specific data and concept drift detection, the three pillars of metrics, logs, and traces, and alerting and on-call practices.
Core ConceptsQuantization represents weights and activations with lower-precision numbers—the best value-for-money acceleration and slimming technique available. This article covers the FP16/BF16/INT8/INT4 trade-offs, PTQ vs QAT, weight vs activation quantization, and GPTQ/AWQ/GGUF in the LLM era.
Core ConceptsA hands-on guide to deploying open-source LLMs (Qwen/Llama and friends) with vLLM behind an OpenAI-compatible API: startup parameter tuning, loading quantized models, tensor parallelism, load testing, and troubleshooting.
Case StudiesA complete, hands-on guide to wrapping a trained PyTorch model in a FastAPI service and shipping it as a Docker image: model export, pre/post-processing wrappers, gunicorn multi-process deployment, health checks, and the usual pitfalls.
Case StudiesThe vLLM paper (SOSP 2023) is essential reading in this field: it brings the OS virtual-memory paging idea to KV cache management, eliminating memory fragmentation and, combined with continuous batching, boosting throughput 2–4×. A section-by-section walkthrough.
PapersA field guide to deployment failures: train/serve preprocessing skew, undetected quantization collapse, GPU memory leaks, launching with no monitoring, big-bang releases... every pitfall gets a symptom → cause → fix, all hard-won lessons, plus a 10-point pre-deployment checklist.
PracticeWalk the complete deployment loop — train → export → serve → load test → monitor — along a minimal viable path. Every step ships with runnable code, a repo layout, common pitfalls, and where to go next, leaving you with a reusable deployment skeleton.
Practice80+ high-frequency interview questions for model deployment roles—inference engine internals, quantization, serving, Kubernetes and containers, monitoring, MLOps, LLM inference, and system design. Each question comes with reference answer points and follow-up directions — the endgame question bank for your job-hunt sprint.
CareerBrowse by section, or search directly
From 2015's "scripts plus hand-copied weights" to today's vLLM continuous batching and cloud-native inference platforms — how far has model deployment come in a decade? This page walks the timeline through four eras, their key tools, pain points, and turning points, so you understand where today's tools came from.
What does a production-grade inference system actually consist of? This page dissects the seven-layer structure — request entry, serving layer, inference engine, resource layer, plus the cross-cutting concerns of data and versioning, observability, and governance — covering each layer's responsibilities, common failure points, and how the layers cooperate.
Model deployment, MLOps, inference, model serving, and inference engine are five terms people constantly mix up. This page pins down each boundary — which one is a subset of which, where each sits in the pipeline — and assembles them into one unified view of the deployment pipeline.
Model deployment knowledge is scattered and unsystematic? This article lays out three learning routes — a job-hunt sprint (about 2 weeks), a systematic deep-dive (about 8 weeks), and a desk reference (dip in as needed) — each with a topic-organized page list and checkpoints.
Model deployment turns a trained model into a production system that keeps serving predictions. This article gives a precise definition, the four elements of deployment, the fundamental difference from training, the unique challenges of model systems (drift, quality degradation), and what a deployment engineer actually solves.
Online, batch, streaming, edge, Serverless—five deployment architectures, each with its own sweet spot. This article provides a full comparison table, a selection decision tree, and a discussion of hybrid deployments.
Beyond quantization, distillation, pruning, and low-rank factorization are three more routes to a smaller model. This article compares how each technique works, where each fits, and how to combine them, so you can pick the right move.
Whether a model runs at all, and how fast, is half decided by the hardware. This article covers the key GPU specs (VRAM, bandwidth, compute), VRAM-to-model-size arithmetic, CPU/GPU/NPU/TPU comparisons, and how to reason about instance selection for inference workloads.
Inference is every forward pass a model makes once it's live. This article covers what inference really is, how it fundamentally differs from training, the latency breakdown of a single inference request, and what inference engines actually do—graph optimization, operator fusion, and kernel selection.
LLM inference is the hottest frontier in model deployment: autoregressive generation, KV cache, continuous batching, speculative decoding, tensor parallelism. This article lays out the GPU memory ledger of LLM serving and every optimization technique that matters.
Model CI/CD is not the same as code CI/CD: you also have to manage model versions, data versions, and quality regressions. This article walks through the deployment pipeline from commit to production — model registry, test gates, canary releases, and rollback.
From PyTorch's .pt to ONNX, TensorRT, TFLite, and GGUF—the model format determines whether you can run efficient inference on the target hardware. This article surveys the mainstream formats, the conversion pipeline, operator compatibility pitfalls, and post-conversion numerical validation.
Serving is the critical leap that turns a model into a callable API. This article explains what an online inference service really is, how to choose between HTTP and gRPC, how to design inputs and outputs, how to wrap preprocessing and postprocessing, and how to manage the model lifecycle.
A model deployment without monitoring is flying blind. This article covers the four golden signals for inference services, ML-specific data and concept drift detection, the three pillars of metrics, logs, and traces, and alerting and on-call practices.
Latency, throughput, and cost form a three-way trade-off in inference serving. This article lays out a performance metric system (P50/P99/QPS/TTFT/TPOT), four layers of optimization (model → engine → service → system), and a capacity planning methodology.
Quantization represents weights and activations with lower-precision numbers—the best value-for-money acceleration and slimming technique available. This article covers the FP16/BF16/INT8/INT4 trade-offs, PTQ vs QAT, weight vs activation quantization, and GPTQ/AWQ/GGUF in the LLM era.
Once a model goes live it is exposed to the public internet, and the security risks follow: model theft, prompt injection, adversarial attacks, data leaks. This article sets out a security baseline for inference services: authentication and authorization, input defenses, privacy protection, and compliance essentials.
Offline batch inference (profile updates, daily report scoring) costs an order of magnitude less than online inference. Build a retryable, checkpoint-resumable batch inference pipeline with Airflow + Python/Spark.
A hands-on guide to deploying open-source LLMs (Qwen/Llama and friends) with vLLM behind an OpenAI-compatible API: startup parameter tuning, loading quantized models, tensor parallelism, load testing, and troubleshooting.
When multiple models or model versions coexist, the gateway owns routing, rate limiting, gradual rollouts, and A/B testing. This hands-on article builds a model gateway with Nginx/Envoy plus application-layer routing, and walks through the complete canary release and A/B experiment workflow.
A complete hands-on guide to hosting multiple models on NVIDIA Triton Inference Server with dynamic batching and concurrent scheduling enabled: model repository layout, writing config.pbtxt, performance metrics, and load-test comparisons.
Online inference for recommender systems is one of the tightest latency-budget deployment scenarios. This guide breaks down how each model in the funnel (recall → coarse ranking → fine ranking → re-ranking) is deployed, how features are fetched, how multiple models are chained, and how the latency budget is allocated.
A complete, hands-on guide to wrapping a trained PyTorch model in a FastAPI service and shipping it as a Docker image: model export, pre/post-processing wrappers, gunicorn multi-process deployment, health checks, and the usual pitfalls.
Serverless is the best fit for low-frequency, bursty, pay-per-call inference. Deploy an inference service on AWS Lambda (or a cloud function), tackle cold starts head-on, and compare cost models against always-on serving.
A hands-on path from optimizing a model into a TensorRT engine to deploying it on edge devices (Jetson): ONNX→TRT conversion, FP16/INT8 accuracy trade-offs, dynamic shapes, and edge deployment under power constraints.
Knowledge distillation has a large model teach a small one — the core technique for trading training cost for inference cost in deployment. This page walks through Hinton's original KD and the BERT-family distillations (DistilBERT, TinyBERT, MiniLM).
The vLLM paper (SOSP 2023) is essential reading in this field: it brings the OS virtual-memory paging idea to KV cache management, eliminating memory fragmentation and, combined with continuous batching, boosting throughput 2–4×. A section-by-section walkthrough.
A paper map of the model deployment field — quantization, distillation, compression, inference serving systems, parallelism and distribution, LLM inference — with representative papers grouped by topic, each with a one-sentence contribution and links to the site's walkthrough pages.
Why should deployment engineers read papers? This page explains how to use the Papers section: the three-pass reading method, recommended reading order by topic, and entry points to every walkthrough.
The model won't fit on one GPU — now what? Tensor parallelism, pipeline parallelism, and ZeRO — this page walks through Megatron-LM, GPipe, and DeepSpeed ZeRO, laying out the parallelism strategies behind large-model training and inference.
Three milestones of LLM quantization: LLM.int8() first ran a 175B model on a single GPU, GPTQ pushed 4-bit quantization to near-lossless, and AWQ used activation awareness to protect the weights that matter. A walkthrough and comparison of all three.
Too many papers, not enough time? This page lays out three reading paths — Beginner (3 papers), Systematic (6 papers), and the LLM Inference track (4 papers) — each with its prerequisites and reading order.
Four key papers on inference serving systems: TensorFlow Serving defined the model serving paradigm, Clipper introduced model selection and adaptive batching, Orca's iteration-level scheduling preceded continuous batching, and Nexus pushed GPU sharding to its limits.
FastAPI, BentoML, Triton, KServe, SageMaker, Volcengine Ark... how do you pick a deployment stack? This guide gives a decision framework based on team size, latency requirements, and model type, plus a scenario-to-solution recommendation matrix.
A field guide to deployment failures: train/serve preprocessing skew, undetected quantization collapse, GPU memory leaks, launching with no monitoring, big-bang releases... every pitfall gets a symptom → cause → fix, all hard-won lessons, plus a 10-point pre-deployment checklist.
Walk the complete deployment loop — train → export → serve → load test → monitor — along a minimal viable path. Every step ships with runnable code, a repo layout, common pitfalls, and where to go next, leaving you with a reusable deployment skeleton.
Three questions you must answer before launch: how much QPS can the service take, what is the P99, and does it need scaling. This guide covers a full load testing methodology: tool selection (wrk/locust/vegeta/ghz/k6), scenario design, interpreting results, and capacity formulas.
How do you slim down and speed up a model before it ships? This page gives a hands-on roadmap — quantize first (start with PTQ), move to QAT/distillation if accuracy falls short, prune when needed — with tools and checkpoints at every step, closing with a decision table.
A complete walkthrough for wiring an inference service into Prometheus + Grafana: custom metrics (request counts/latency histograms/error rates), GPU metric collection, SLO burn-rate alert rules, and logs with tracing, plus a 30-day post-launch observation checklist.
Model releases differ from code releases: quality must be validated in production, and rollbacks must take seconds. This guide compares canary, blue/green, and A/B releases, walks through each process step by step, and closes with a pre-release checklist.
Which bodies of knowledge hide behind JD keywords like K8s, ONNX, TensorRT, vLLM, and performance tuning? This page maps high-frequency JD keywords to the relevant pages on this site, with recommended mastery depth, so you know what to learn and how deep to go.
What roles exist in model deployment? What do ML platform engineers, inference engine engineers, MLOps engineers, and AI Infra engineers actually do, and what do they require? This page maps the job landscape, compares responsibilities, covers industry trends, and shows how to use this module to prepare for your job search.
80+ high-frequency interview questions for model deployment roles—inference engine internals, quantization, serving, Kubernetes and containers, monitoring, MLOps, LLM inference, and system design. Each question comes with reference answer points and follow-up directions — the endgame question bank for your job-hunt sprint.
What do real model deployment job descriptions look like? This page compiles inference / MLOps / ML platform roles currently open at 15+ companies in China and abroad, breaks down the core requirements line by line, and marks estimated salary ranges (for reference only, not official).
What should a deployment engineer's resume highlight? This page covers STAR-format project writing, formulas for quantifying results (latency/QPS/cost/availability), keyword-matching strategy, and how to build a portfolio project — plus anti-patterns to avoid.
Curated resources for model deployment: official documentation, classic courses, high-quality blogs, and communities—organized by purpose with a one-line review for each entry, so you never have to dredge the internet blind.
Core terminology for model deployment: fundamentals, models and formats, compression and optimization, serving and architecture, hardware and performance, monitoring and operations, and LLM inference—precise definitions and distinctions for 60+ terms.
A systems-knowledge cheat sheet every model deployment engineer needs: Linux, containers and Docker, Kubernetes, networking and process models, GPU environments, and performance tools—at your fingertips whenever you're troubleshooting a deployment.
A panoramic profile of model deployment tools: inference engines, serving frameworks, monitoring and experiment management, cloud platforms, and CI/CD—organized by purpose into tables with positioning, use cases, and difficulty ratings for selection reference.
