INFERENCE ACCELERATION HANDBOOK
A systematic knowledge base, from weights to serving — Performance & Bottlenecks · Quantization & Compression · Kernels & Graph Optimization · Serving & Orchestration · Engine Case Studies · Papers & Career Paths
50+ pages that form not a library to read cover to cover, but routes you can mix and match
Interviewing for inference acceleration roles within 1–3 months: go straight to JD benchmarking, resume rewriting, and the interview question bank — every page maps to what big tech actually tests.
See the path →With time to build a real foundation: work from the guide through hands-on practice week by week, each week with a clear checkpoint.
See the path →For working inference engineers who won't read cover to cover: a problem → page index for quick lookups — use it as a reference book.
See the path →If you only read ten pages, read these
Inference is all the engineering that turns trained model weights into a production-servable system. This page gives a three-level definition, the fundamental differences from training, the four layers of optimization, a hand-written minimal inference system (with KV cache), and three empirical speedup milestones.
Getting StartedThe GPU is the workhorse hardware of LLM inference — the architecture of SMs, warps, Tensor Cores, HBM, and L2 cache dictates every optimization direction. This article covers the GPU programming model, compute and bandwidth specs across generations, a performance-tuning checklist (occupancy, coalesced access, bank conflicts, async memcpy, pipelining), and the cuDNN/cuBLAS/CUTLASS libraries.
Core ConceptsKernel fusion merges multiple elementwise/reduction operators into a single kernel, avoiding HBM round trips for intermediate tensors — the core technique of LLM inference optimization. This article covers classic fusion patterns, FlashAttention v1/v2/v3's SRAM tiling, FlashInfer's block sparsity, the Triton/CUDA programming models, and how to judge fusion boundaries.
Core ConceptsThe three core metrics of an inference system — latency (TTFT/TPOT/E2E), throughput (tokens/s, QPS), and concurrency — and the trade-offs among them. This article covers Little's Law, the divergent latency characteristics of the prefill and decode phases, the latency-throughput curve, and measurement methodology.
Core ConceptsQuantization is the core technique that compresses FP16/BF16 weights and activations down to INT8/INT4/FP8/FP4 — trading a small amount of accuracy for 2-8× inference speedups and memory savings. This article covers the linear quantization formula, quantization granularity, PTQ vs. QAT, calibration methods, and the mainstream algorithm families GPTQ/AWQ/SmoothQuant/FP8/FP4.
Core ConceptsONNX Runtime takes "model as a file" to the extreme — a single .onnx file runs on CPU, CUDA, DirectML, CoreML, and QNN, powered by the Execution Provider mechanism and a rigorous graph optimization pipeline. This article dissects the ONNX IR, the EP mechanism, the GRPO optimizer, and the cross-framework toolchain.
Case StudiesA small draft model predicts k tokens; the large model verifies them all in a single forward pass and accepts every correct one — the most elegant lossless 2–6× acceleration for the LLM decode phase. This article dissects three generations of speculative decoding, the mathematical losslessness guarantee, and the boundary conditions.
Case StudiesNVIDIA's own GPU inference optimization engine, which turns the "general-purpose compute" of training frameworks into "extreme throughput on the GPU" through graph fusion, kernel auto-tuning, and INT8/FP8 quantization. This article dissects the three-stage Builder/Engine/Runtime pipeline, INT8 calibration, the Plugin interface, and torch_tensorrt integration.
Case StudiesNVIDIA's LLM-specific extension on top of TensorRT, integrating In-Flight Batching, Paged KV Cache, FlashAttention, FP8, and more — the flagship stack for LLM inference on H100. This article dissects the build flow, tuning switches, the comparison with vLLM, and Triton deployment.
Case StudiesThe LLM inference engine open-sourced by UC Berkeley in 2023. PagedAttention pages the KV cache like virtual memory, and with continuous batching it lifts Llama-65B throughput by an order of magnitude. This article dissects PagedAttention, prefix caching, quantization support, and the comparison with SGLang/TGI.
Case StudiesFrom FlashAttention to DeepSeek-V3, a close reading of the ten classic papers that changed the inference acceleration and deployment field — background, mechanism, experiments, implications — plus the three-pass method for reading papers and the common laws behind the ten papers, to help you build a coordinate system for reading inference papers.
Paper Deep DivesThe inference-acceleration frontier shifts every week. This page shows how to track research with arXiv, top systems conferences, and open-source repos; breaks down six main lines — operators, serving systems, speculative decoding, quantization, architecture innovation, and hardware generations; honestly assesses the unsolved problems; and helps you choose a direction to go deep.
Paper Deep DivesWhy should you read papers even if you work on inference acceleration and deployment? This page explains the distinct value of three kinds of papers, offers three entry routes by goal, introduces the five-part navigation of this papers section, and closes with the single most important piece of advice for reading papers.
Paper Deep DivesAccuracy not aligned before and after quantization, preprocessing inconsistent with training, single-batch latency mistaken for production, KV cache misestimation, GPU utilization as the only metric, speculative decoding at large batch... This checklist covers 15 high-frequency pitfalls in inference deployment, each with symptom, root cause, fix, and related page links, plus a tickable self-check list.
Hands-On GuidesWalk through a three-stage progressive path with Llama-2-7B — from raw PyTorch inference, to swapping in vLLM, to AWQ quantization + EAGLE speculative decoding + Triton deployment. Each stage ships runnable commands, observed metrics, a directory layout, and performance comparisons: the most concentrated end-to-end hands-on project in inference deployment.
Hands-On GuidesHow do you measure, report, and cheat-proof inference optimization numbers? This article presents a methodology for LLM inference benchmarking: what to measure (throughput / TTFT / TPOT / p99 / memory), what to measure with (vLLM offline / mlperf-llm / sglang bench / lm-eval-harness), how to control variables, and how to write the report — with an anti-cheat checklist and a report template.
Hands-On GuidesHow do you choose among vLLM, SGLang, TensorRT-LLM, TGI, Triton, ONNX Runtime, OpenVINO, llama.cpp, and TFLite? Starting from five dimensions — model type, hardware, deployment form, performance target, and team capability — this article gives a nine-engine comparison table and selection decisions for three major scenarios: LLM online serving, classic model serving, and on-device.
Hands-On GuidesThe full interview landscape for inference acceleration roles with high-frequency question walkthroughs — ten categories, 28 questions covering performance metrics, quantization, KV cache, batching, kernels, serving & orchestration, speculative decoding, distributed inference, hardware, and system design; includes a whiteboard KV cache draft, scripts for saying "I don't know," and 1-week / 1-month sprint checklists.
CareersA panoramic view of inference acceleration roles — from Silicon Valley to China. Responsibilities, competency models, and salary bands for seven role families (engine engineer, systems engineer, optimization engineer, infra/MLOps engineer, kernel engineer, edge engineer, framework R&D), plus three persona profiles and a map of the four career-module pages.
CareersCore terminology for inference acceleration — over 70 precisely defined and cross-examined terms across seven groups: fundamentals, performance & metrics, memory & compute, quantization & compression, operators & graphs, systems & serving, and hardware.
ResourcesHardware architecture for inference acceleration — GPU/SM/Tensor Core, a three-generation comparison of NVIDIA datacenter cards (A100/H100/H200/B200) on memory, bandwidth, compute, and power, consumer GPUs, AMD MI300X, Apple Silicon, Ascend 910B, Intel AMX, NPUs, and the training-vs-inference selection logic.
ResourcesBrowse by module, or search directly
From 2010s CPU inference and OpenVINO, to the rise of GPU/cuDNN, to the Transformer and LLM inference revolution, and on to vLLM/PagedAttention, FP8, and speculative decoding. This page traces fifteen years of inference acceleration on a single timeline and the technical reasons behind each leap.
The complete skeleton of a production-grade LLM inference system across six layers — application, orchestration, engine, operator, model, and hardware. This page is the content map of the whole site; each layer points to the corresponding deep-dive articles, and after reading it you will have the global view of inference engineering.
Training counts FLOPs and activations; inference is memory-access-heavy, batching-sensitive, and requires a KV cache. Fine-tuning is a lightweight epilogue of training. This page draws the three apart across seven dimensions — compute, memory access, batching, KV cache, precision, hardware, and where fine-tuning sits.
The 50+ pages on this site are not a library to read cover to cover — they are roadmaps you combine as needed. There is a job-sprint track for anyone interviewing for an inference/deployment role within three months, an eight-week systematic track for those who want a solid foundation, and a desk-reference index that maps "problem -> page" for practicing deployment engineers.
Inference is all the engineering that turns trained model weights into a production-servable system. This page gives a three-level definition, the fundamental differences from training, the four layers of optimization, a hand-written minimal inference system (with KV cache), and three empirical speedup milestones.
Batching is the core performance knob of GPU inference — static batching waits until a batch is full (high latency), dynamic batching is a compromise, and continuous batching can swap requests in and out at every step (the innovation of vLLM/SGLang). This article covers scheduling policies, prefill/decode disaggregated scheduling, and the vLLM scheduler mechanism.
The computation graph (IR) is the hub of an inference engine — constant folding, dead-code elimination, operator fusion, layout propagation, CUDA graph capture, and prefix caching are all graph-level optimizations. This article covers the ONNX/MLIR/XLA HLO IRs, static vs. dynamic shapes, and the TVM Relax/Inductor/XLA/Triton compiler stack.
The GPU is the workhorse hardware of LLM inference — the architecture of SMs, warps, Tensor Cores, HBM, and L2 cache dictates every optimization direction. This article covers the GPU programming model, compute and bandwidth specs across generations, a performance-tuning checklist (occupancy, coalesced access, bank conflicts, async memcpy, pipelining), and the cuDNN/cuBLAS/CUTLASS libraries.
Kernel fusion merges multiple elementwise/reduction operators into a single kernel, avoiding HBM round trips for intermediate tensors — the core technique of LLM inference optimization. This article covers classic fusion patterns, FlashAttention v1/v2/v3's SRAM tiling, FlashInfer's block sparsity, the Triton/CUDA programming models, and how to judge fusion boundaries.
Knowledge distillation (Hinton 2015) lets a large model (teacher) teach a small model (student), transferring "dark knowledge" through soft labels. This article covers the three types of supervision (soft/hard/feature), the role of temperature T, LLM distillation (distilling small models from GPT-4, MiniLLM white-box distillation), and the role of distillation in the "compression trio."
The three core metrics of an inference system — latency (TTFT/TPOT/E2E), throughput (tokens/s, QPS), and concurrency — and the trade-offs among them. This article covers Little's Law, the divergent latency characteristics of the prefill and decode phases, the latency-throughput curve, and measurement methodology.
Quantization is the core technique that compresses FP16/BF16 weights and activations down to INT8/INT4/FP8/FP4 — trading a small amount of accuracy for 2-8× inference speedups and memory savings. This article covers the linear quantization formula, quantization granularity, PTQ vs. QAT, calibration methods, and the mainstream algorithm families GPTQ/AWQ/SmoothQuant/FP8/FP4.
Turning a model from a notebook into a stable production service — the layered architecture of inference serving (API gateway / inference engine / model repository / load balancing / health checks), deployment topologies, mainstream frameworks (Triton / vLLM / TGI / SGLang / Ray Serve), multi-model routing, autoscaling, monitoring metrics, and the MLOps loop.
Pruning removes unimportant weights or structures to make models smaller and faster. This article covers unstructured vs. structured pruning, the Lottery Ticket Hypothesis, magnitude/iterative/Taylor/movement pruning algorithms, NVIDIA 2:4 structured sparsity, and Wanda/LLM-Pruner in the LLM era.
LLM inference in the decode phase is throttled almost entirely by GPU memory bandwidth — every generated token requires reading the entire model weights from HBM once. This article covers the memory hierarchy (HBM/L2/SRAM), bandwidth specs across GPU generations, the KV cache memory formula, and how to identify and fight the memory-bound regime.
The Roofline model is a unified framework for analyzing GPU operator performance — attainable FLOPS = min(peak FLOPS, bandwidth × arithmetic_intensity). This article draws a Roofline chart, explains arithmetic intensity, the ridge point, and the memory-bound vs. compute-bound classification, and uses it to diagnose LLM inference bottlenecks.
Weight-only quantization compresses LLM weights to INT4/INT8 while keeping activations in FP16/BF16 — purpose-built for the decode-phase bandwidth wall. This article covers GPTQ's Hessian-based second-order compensation, AWQ's salient-weight protection, the Marlin kernel's extreme fusion, and how to choose among W8A16/W4A16/W4A8 combinations.
When an LLM no longer fits into a single GPU's memory, the model must be sharded across cards — tensor parallelism splits by head, pipeline parallelism splits by layer, and expert parallelism splits by MoE routing. This article dissects the three sharding strategies, communication costs, mainstream implementations, and cross-node deployment.
A pure C++ LLM inference framework. With the GGUF file format and k-quants block quantization, it squeezes Llama-70B into a MacBook Pro running at 10 tokens/s — the de facto first choice for on-device LLMs and "running large models on consumer hardware."
iOS CoreML/Metal, Android NNAPI/Hexagon, cross-platform PyTorch Mobile/TFLite — squeezing models into the triple constraints of memory, power, and thermals on iPhone and Android. This article dissects model conversion, quantization, NPU acceleration, and the state of on-mobile LLMs.
ONNX Runtime takes "model as a file" to the extreme — a single .onnx file runs on CPU, CUDA, DirectML, CoreML, and QNN, powered by the Execution Provider mechanism and a rigorous graph optimization pipeline. This article dissects the ONNX IR, the EP mechanism, the GRPO optimizer, and the cross-framework toolchain.
Intel's own CPU/iGPU inference optimization engine — convert to IR with the Model Optimizer, deeply tuned AVX512/AMX instructions, and one-stop INT8 quantization with NNCF. It is the zero-cost, first-choice option for inference on Intel servers.
A small draft model predicts k tokens; the large model verifies them all in a single forward pass and accepts every correct one — the most elegant lossless 2–6× acceleration for the LLM decode phase. This article dissects three generations of speculative decoding, the mathematical losslessness guarantee, and the boundary conditions.
NVIDIA's own GPU inference optimization engine, which turns the "general-purpose compute" of training frameworks into "extreme throughput on the GPU" through graph fusion, kernel auto-tuning, and INT8/FP8 quantization. This article dissects the three-stage Builder/Engine/Runtime pipeline, INT8 calibration, the Plugin interface, and torch_tensorrt integration.
NVIDIA's LLM-specific extension on top of TensorRT, integrating In-Flight Batching, Paged KV Cache, FlashAttention, FP8, and more — the flagship stack for LLM inference on H100. This article dissects the build flow, tuning switches, the comparison with vLLM, and Triton deployment.
NVIDIA's open-source multi-framework, multi-model, multi-backend inference server — the de facto standard for taking LLMs and traditional models to production. This article dissects the Backend concept, config.pbtxt, concurrent execution, dynamic batching, the model repository, Prometheus metrics, and KServe integration.
The LLM inference engine open-sourced by UC Berkeley in 2023. PagedAttention pages the KV cache like virtual memory, and with continuous batching it lifts Llama-65B throughput by an order of magnitude. This article dissects PagedAttention, prefix caching, quantization support, and the comparison with SGLang/TGI.
From FlashAttention to DeepSeek-V3, a close reading of the ten classic papers that changed the inference acceleration and deployment field — background, mechanism, experiments, implications — plus the three-pass method for reading papers and the common laws behind the ten papers, to help you build a coordinate system for reading inference papers.
The inference-acceleration frontier shifts every week. This page shows how to track research with arXiv, top systems conferences, and open-source repos; breaks down six main lines — operators, serving systems, speculative decoding, quantization, architecture innovation, and hardware generations; honestly assesses the unsolved problems; and helps you choose a direction to go deep.
A searchable map of the key inference-acceleration and deployment papers from 2014-2025, grouped by theme and covering six areas — operator optimization, quantization, serving systems, speculative decoding, distributed training/inference, and on-device deployment — with a paradigm-shift timeline and a task-based paper index.
A methodology handbook for reading inference papers: how to read speedups and profile figures, how to judge a paper's quality with red-flag signals, and point-by-point answers to 12 high-frequency questions — what to do when you cannot understand, cannot remember, cannot find related papers, or cannot finish reading.
Inference papers are vast and deep, and diving in blindly only leads to burnout. This page gives three paths — the engineer-onboarding line, the systematic deep-dive line, and the researcher line — plus the three-pass method, so you can pick a route by goal and read at a steady rhythm.
Why should you read papers even if you work on inference acceleration and deployment? This page explains the distinct value of three kinds of papers, offers three entry routes by goal, introduces the five-part navigation of this papers section, and closes with the single most important piece of advice for reading papers.
Accuracy not aligned before and after quantization, preprocessing inconsistent with training, single-batch latency mistaken for production, KV cache misestimation, GPU utilization as the only metric, speculative decoding at large batch... This checklist covers 15 high-frequency pitfalls in inference deployment, each with symptom, root cause, fix, and related page links, plus a tickable self-check list.
Walk through a three-stage progressive path with Llama-2-7B — from raw PyTorch inference, to swapping in vLLM, to AWQ quantization + EAGLE speculative decoding + Triton deployment. Each stage ships runnable commands, observed metrics, a directory layout, and performance comparisons: the most concentrated end-to-end hands-on project in inference deployment.
Inference deployment projects rarely fail because the engine isn't advanced enough — they die from design mistakes. This article distills twelve deployment design principles: latency-first vs throughput-first, memory budget, streaming beats waiting, three-layer bottleneck checks, activation outliers before quantization, speculative decoding vs batch size, 5% gray releases, KV hit-rate monitoring, failure drills, end-to-end benchmarks, and more.
How do you measure, report, and cheat-proof inference optimization numbers? This article presents a methodology for LLM inference benchmarking: what to measure (throughput / TTFT / TPOT / p99 / memory), what to measure with (vLLM offline / mlperf-llm / sglang bench / lm-eval-harness), how to control variables, and how to write the report — with an anti-cheat checklist and a report template.
How do you choose among vLLM, SGLang, TensorRT-LLM, TGI, Triton, ONNX Runtime, OpenVINO, llama.cpp, and TFLite? Starting from five dimensions — model type, hardware, deployment form, performance target, and team capability — this article gives a nine-engine comparison table and selection decisions for three major scenarios: LLM online serving, classic model serving, and on-device.
Five portfolio-worthy projects for inference and deployment roles: a RAG inference service, multi-model routing, a streaming chat API, a quantization comparison benchmark platform, and an on-device LLM app. Each project comes with motivation, tech stack, deliverables, and selling points you can talk about.
Break the v1→v2→v3 deployment optimization into six versions (v0 baseline → v1 KV cache → v2 continuous batching → v3 INT4 → v4 speculative decoding → v5 Triton + autoscale), adding exactly one optimization per version, with diff-style code and the corresponding benchmark data so you can learn them one by one.
A tuning checklist and performance optimization playbook for LLM inference: detailed coverage of the eight core parameters — max_num_seqs, max_model_len, gpu_memory_utilization, block_size, swap_space, prefix_caching, chunked_prefill, tensor_parallel_size — plus bottleneck diagnosis with nvidia-smi, Nsight Systems, and PyTorch Profiler, and response strategies for compute-bound vs memory-bound workloads.
Turn skill phrases from inference & deployment JDs — "proficient in vLLM," "familiar with CUDA," "quantization experience" — into a knowledge-point checklist you can review item by item. A three-layer competency structure (L1 fundamentals / L2 topics / L3 advanced) plus a five-level self-assessment scale, from "never heard of it" to "can teach it," generates your personal catch-up roadmap.
The full interview landscape for inference acceleration roles with high-frequency question walkthroughs — ten categories, 28 questions covering performance metrics, quantization, KV cache, batching, kernels, serving & orchestration, speculative decoding, distributed inference, hardware, and system design; includes a whiteboard KV cache draft, scripts for saying "I don't know," and 1-week / 1-month sprint checklists.
A curated dissection of real JDs for open inference acceleration roles at major companies — covering ByteDance Doubao, Alibaba PAI, Huawei Ascend, OpenAI, Anthropic, Meta, NVIDIA, Groq and 14 roles in total. Extracts a high-frequency skill-word table mapped to this site's learning pages, helping you back-solve a study plan from JDs.
A panoramic view of inference acceleration roles — from Silicon Valley to China. Responsibilities, competency models, and salary bands for seven role families (engine engineer, systems engineer, optimization engineer, infra/MLOps engineer, kernel engineer, edge engineer, framework R&D), plus three persona profiles and a map of the four career-module pages.
A resume for an inference & deployment role is not a list of experiences — it's an evidence chain of "performance data + systems understanding + engineering ability." This page gives you a JD skills-benchmarking method, a deployment-flavored STAR format, 6 project rewrites, a layered tech-stack strategy, ten fatal counterexamples, and a ready-to-use resume template skeleton.
Benchmark data and tool profiles for inference acceleration — MLPerf Inference 4.1/5.0, public benchmarks of vLLM/SGLang/TensorRT-LLM, LLM throughput and latency across GPUs, INT8/INT4/FP8 comparisons, engine comparisons, and a quick reference of benchmarking tools.
From vLLM to TensorRT-LLM, from FlashAttention to PagedAttention, from CUDA Mode to CMU 15-819 — a topic-organized map of learning resources for LLM inference acceleration and deployment, with every official link verified.
Core terminology for inference acceleration — over 70 precisely defined and cross-examined terms across seven groups: fundamentals, performance & metrics, memory & compute, quantization & compression, operators & graphs, systems & serving, and hardware.
Hardware architecture for inference acceleration — GPU/SM/Tensor Core, a three-generation comparison of NVIDIA datacenter cards (A100/H100/H200/B200) on memory, bandwidth, compute, and power, consumer GPUs, AMD MI300X, Apple Silicon, Ascend 910B, Intel AMX, NPUs, and the training-vs-inference selection logic.
