Inference: From Forward Pass to Inference Engines

The one-sentence definition: inference is the single forward pass a model executes each time it receives an input after deployment—freezing the trained weights as a read-only asset and computing outputs for new inputs.

Industry insight: for most companies, inference eventually consumes more compute than training. A recommendation model trained for three months internally gets called hundreds of millions of times a day once live; OpenAI once disclosed that inference accounts for over 40% of its total infrastructure spend, and with the LLM boom, inference's share of total GPU purchases will only climb. That's why the core question for a deployment engineer isn't "getting the model to run" (that's just an extension of training)—it's "getting it to run correctly, fast, and cheaply." The forward pass happens only once per request, so it has to be done right and done to the limit.

1. Inference: A Read-Only Forward Pass

1.1 The Fundamental Differences Between Training and Inference

Training and inference share the same forward-pass math, but their engineering constraints are completely different. Internalizing this comparison table is the prerequisite for understanding every deployment optimization that follows:

Dimension	Training	Inference
Propagation direction	Forward + backward	Forward only
Weight state	Continuously updated (writable)	Frozen (read-only)
Batch size	Large (64–1024), throughput-driven	Variable: mostly 1 online, large offline
Latency sensitivity	Low—minutes are acceptable	Extreme—P99 must be milliseconds
Numerical precision	FP32 / BF16 (gradient precision required)	FP16 / INT8 / INT4 all fine
Memory needs	Weights + gradients + optimizer states + activations (~16–20× parameter count)	Weights + activations (~1.2–2× parameter count)
Core bottleneck	Compute (FLOPs)	Bandwidth (memory-bound, see below)
Correctness criterion	Loss converges	Output matches expectations (sometimes reproducibility too)
Cost of failure	Retrain	Production incident

A Common VRAM Misconception

Training a 7B model takes 200GB+ of VRAM (gradients and optimizer states included), but deploying it takes only about 14GB (FP16 weights)—"it needed that much to train, so deployment must need even more" is wrong. Inference stores no gradients, and frozen weights can be quantized and compressed, so the memory pressure is actually an order of magnitude smaller. See GPUs and Hardware Selection for details.

1.2 Inference Does Exactly One Thing: Compute y = f(x)

No matter how complex the model, the inference computation graph just transforms the input into the output layer by layer. Take a BERT classification model:

Input token sequence
   │  embedding lookup
   ▼
12 Transformer encoder layers (each with Self-Attention + FFN)
   │  LayerNorm / residual
   ▼
[CLS] vector → linear layer → softmax
   ▼
Class probability distribution (the inference output)

Three key engineering implications:

1. Weights are read-only: once deployed, weights never change, so you can pre-apply graph optimization and operator fusion, or even compile the model into a dedicated binary (like a TensorRT engine) to push runtime overhead down to the minimum.
2. Outputs are cacheable: during training, the same input appears only once; during inference, the same input (say, the embedding of a popular product) gets queried over and over—which makes result caching one of the most valuable optimizations for an inference service.
3. Input-independent work can be precomputed: anything that doesn't depend on the input (constant folding, weight repacking) can be done at deployment time, leaving the runtime to execute only the essential operators.

1.3 The Three Forms of Inference

By invocation pattern, inference falls into three classes; Deployment Architecture Patterns expands on this later:

• Online inference: returns immediately when a request arrives, millisecond latency—recommendations, translation, chat. The latency breakdown in Section 3 of this article is about this form.
• Offline (batch) inference: processes a large volume of data in one go, hour-scale—user profiles, daily reports.
• Streaming inference: event-driven, record-at-a-time—real-time risk control, fraud detection.

2. The Latency Breakdown of a Single Online Inference

2.1 The End-to-End Timeline: Latency Hides in the Non-Model Parts

Plenty of teams spend the better part of a year optimizing their model, run a load test, and find that P99 hasn't budged—the cause is usually that the bottleneck was never the model. Here's the latency breakdown of a typical online inference request:

  Client                API Gateway            Inference Service      GPU
   │  ① serialize request (JSON)   │                       │                │
   │──────────────────────────────►│  ② network transit (RTT)               │
   │                               │───────────────────────────────────────►│
   │                               │  ③ deserialize / validate params       │
   │                               │  ④ preprocess (feature joins/normalize)│
   │                               │  ⑤ queue wait (batching/scheduling)    │──► ⑥ forward pass
   │                               │  ⑦ postprocess (softmax/argmax)        │
   │                               │  ⑧ serialize response                  │
   │◄─────────────────────────────────────────────────────────────────────────│
   │  ⑨ network return (RTT)       │                       │                │

Take a typical recommendation-scenario figure (single T4, single request, batch=1):

Stage	Typical time	Share	Notes
① Serialization + network (once each way)	0.5–2 ms	5–10%	Grows noticeably with large payloads
④ Preprocessing	0.5–3 ms	5–15%	Feature joins, normalization, embedding lookup
⑤ Queue wait	0–100 ms	Varies the most	Dominant latency source at high concurrency
⑥ Model forward pass	2–50 ms	40–80%	At batch=1 it's actually bandwidth-limited
⑦ Postprocessing	0.1–0.5 ms	<5%	softmax / filtering / truncation

Measure First, Then Optimize

The first rule of latency optimization is instrument before you touch anything: add OpenTelemetry spans to every stage ②–⑧ (see Monitoring and Observability). It's common to discover that "the model accounts for 40% of latency and queuing another 40%"—in which case you should optimize scheduling, not the model.

2.2 The Two Physical Quantities That Determine Forward-Pass Latency: Compute and Bandwidth

How fast the model forward pass (⑥) runs is governed by two hardware quantities:

• Compute (FLOPs): how many floating-point operations per second, measured in TFLOPS.
• Memory bandwidth: how much data per second can be read from / written to VRAM, measured in GB/s.

For every operator there's an "arithmetic intensity" ratio: how many operations per byte of data. Below the hardware's balance point (the ridge point in the Roofline model), an operator is memory-bound; above it, compute-bound.

Inference is almost always memory-bound: one matrix multiply, [1, 4096] × [4096, 4096], needs to read 64MB of weights while performing only 3.4 billion operations. An A100's FP16 peak is ~312 TFLOPS with ~2TB/s of bandwidth, putting its balance point at ~156 FLOP/byte—while this computation has only 52 FLOP/byte. Nearly all the time goes into moving weights from VRAM into the compute units. That's where the saying "inference eats bandwidth, not compute" comes from, and it's the fundamental reason quantization delivers near-linear speedups: INT8 weights take half the bytes, so the transfer time is nearly halved.

2.3 The Harsh Reality of batch=1

Online inference defaults to batch=1 (one forward pass per request), which means there isn't enough parallel work to keep the GPU fed—GPU utilization often sits in the single-digit percentages. This is the biggest engineering gap between training and inference: training engineers chase "saturating the GPU," while inference engineers chase exactly the opposite—"minimizing latency at tiny batch sizes." The fix is dynamic batching and sharing capacity across requests; see Performance Optimization and Capacity Planning.

3. What Inference Engines Actually Do

3.1 Why Running PyTorch Directly Isn't Fast Enough

A model "runs" in PyTorch, but for deployment you usually pass it through an inference engine (ONNX Runtime, TensorRT, OpenVINO, TFLite). The reason is that PyTorch's eager mode makes many concessions for training's sake:

• Every operator is dispatched and scheduled dynamically, with heavy Python interpreter overhead;
• Operator boundaries leave many intermediate tensor allocations in place;
• No memory reuse—frequent VRAM allocations and frees;
• Generic kernels aren't specialized for specific shapes.

An inference engine eliminates all of these costs that "don't matter during training but are expensive at inference." Its work breaks down into four steps.

3.2 Graph Optimization

Take the entire computation graph first, then apply input-independent transformations:

Original graph                         Optimized
┌──────────────────┐        ┌──────────────────┐
│ conv ── bn ── relu│  fuse  │ conv+bn+relu      │
│ add ─────────────│  ───►  │ (single kernel)   │
│ mul(const)       │  fold  │ constant 8.0      │
│ softmax(const in)│  ───►  │ precomputed 0.5   │
└──────────────────┘        └──────────────────┘

The concrete techniques:

• Operator fusion: merge conv + batch_norm + relu into a single kernel, eliminating the write-out and re-read of intermediate tensors. In memory-bound scenarios, the bandwidth that fusion saves often is the speedup.
• Constant folding: computations that depend only on weights are precomputed at deployment—for example, BatchNorm can be folded into the convolution's weights and bias at inference time (standard industry practice; torch.quantization.fuse_modules in PyTorch does exactly this).
• Memory planning: analyze tensor lifetimes up front and reuse the same memory regions, avoiding repeated malloc calls at runtime. Typical examples: ONNX Runtime's memory arena and TensorRT's memory pool.
• Dead code and redundancy elimination: strip training-only operators (dropout, gradient-related nodes).

3.3 Kernel Selection: One Operator, Dozens of Implementations

The same matrix multiply has wildly different fastest implementations depending on shape, layout, and hardware. Engines perform kernel autotuning at deployment: benchmark a set of candidate kernels, pick the fastest, and cache the choice. Typical sources of variation:

• Input shape: [1, 4096]×[4096, 4096] (common in inference) and [64, 4096]×[4096, 4096] (common in training) call for completely different tile schemes.
• Data layout: NCHW vs NHWC can differ by 30%+ on some hardware; TensorRT will even insert layout transforms on demand.
• Tensor core availability: whether FP16 runs on tensor cores or CUDA cores is a 5–10× performance difference.
• Tile size and parallelization strategy: how an operator is split across thread blocks and warps.

That's the answer to "why are engines 2–10× faster than the original framework"—the math didn't change; the same math got an execution schedule much closer to the hardware. TensorRT in FP16 can reach 1/3 to 1/5 of PyTorch's latency on ResNet-50, which is exactly graph optimization + kernel specialization + compilation stacked on top of each other.

3.4 The Price Engines Charge: Portability for Performance

The more aggressive an engine's optimizations, the stronger its assumptions about input shapes, model structure, and hardware model. Therefore:

• Dynamic shapes are constrained: TensorRT supports dynamic shapes, but only within a preset range (e.g., batch 1–32); exceeding it means re-optimizing.
• Build once, run on one target: an engine built for an A100 won't run on a T4.
• Precision may drift: operator reordering, low-precision kernels, and different accumulation orders make results differ slightly from the training framework.

For format-level choices, see Model Formats and Conversion; for debugging "why the engine's results differ from the original framework," see Common Pitfalls and Anti-Patterns.

4. CPU Inference vs GPU Inference

4.1 They're Physically Different Machines

Dimension	CPU	GPU
Cores	Dozens (large cores)	Thousands to tens of thousands (small cores)
Cache	Large (tens of MB of L2/L3)	Small (relies mainly on VRAM bandwidth)
Good at	Serial logic, branching, small batches	Massively parallel matrix math
Typical latency	Fast per token, limited throughput	Extremely high throughput once batches fill
Precision	FP32 / INT8 (AVX-512, AMX)	FP16/INT8/INT4 + tensor cores

4.2 When CPU Is the Right Choice

Low QPS + small models is the CPU's comfort zone: a 100MB classification model runs at 5–10ms on CPU versus 2–3ms on GPU—spending serious money on a GPU to save those few milliseconds doesn't pay. In the LLM era, CPU inference has become genuinely usable thanks to llama.cpp's optimizations (AVX2/AVX-512, oneDNN): a 7B quantized model runs at roughly 5–10 tokens/s on an 8-core server, good enough for personal and internal tooling.

4.3 A Simple Decision Rule

Forward pass < 20ms and QPS < 50 and model < 1GB   → CPU is enough
Pure matrix math, batchable, latency-sensitive     → GPU
Edge devices / no GPU available                    → CPU or NPU (see /concepts/hardware)

5. Where Precision Differences Come From

5.1 Same Model, Different Engines—Why Aren't the Results Identical?

This is a confusion every deployment engineer runs into eventually: ONNX Runtime and PyTorch run the same model, and the outputs differ by ~1e-4. Three sources:

1. Different operator implementations: frameworks use different floating-point evaluation orders and intermediate precisions for the same math (layernorm, softmax)—FP32 accumulation vs FP16 accumulation.
2. Fusion changes intermediate precision: after conv+bn fusion, the intermediate tensor never touches VRAM and may stay FP16 throughout—saving the precision loss can also change the result.
3. Low-precision kernels: FP16/INT8 math carries rounding error, and that error accumulates through the network.

5.2 The Standard Way to Validate Numerics

• Set thresholds: rules of thumb—against an FP32 baseline, a max absolute output error < 1e-3 (CV tasks) and cosine similarity > 0.999 are usually acceptable; after INT8 quantization, require the metric drop to be < 1% relative to the baseline.
• Sample with real data: don't fabricate samples from the training set—sample data drawn from the same distribution as production; see the numerical validation section of Model Formats and Conversion.
• Be accountable to business metrics: the final comparison should be task metrics like AUC, BLEU, or Rouge, not bit-for-bit equality.

Trade-offs

Decision point	Option A	Option B	How to choose
Run on framework directly vs through an engine	Fast, easy	2–10× faster, compilable	Always use an engine in production; running directly is fine in dev
General engine vs specialized engine	ONNX Runtime/OpenVINO	TensorRT/TFLite	Specialized for hardware limits, general for portability
High precision vs low precision	FP16/FP32, worry-free	INT8/INT4, fast but needs validation	Ship FP16 first, quantize if latency falls short (see Quantization)
Small batch or large batch	Latency-optimized, GPU idles	Throughput-optimized, latency suffers	Online: small batch + batching; offline: large batch

One-line summary: the engineering essence of inference is turning "one reproducible forward pass" into "the fastest possible one on the hardware"—first make sure the math is right, then squeeze out performance layer by layer.

Further Reading

• Model Formats and Conversion — from .pt to ONNX/TensorRT, the entry point to engine optimization
• GPUs and Hardware Selection — how compute and bandwidth bound the forward pass
• Performance Optimization and Capacity Planning — the latency, throughput, cost triangle
• Common Pitfalls and Anti-Patterns — numerical mismatches, shape problems, and other field wounds
• Quantization — trading bit width for bandwidth, the king of inference speedup levers

References

• ONNX Runtime — Graph Optimizations
• NVIDIA TensorRT Developer Guide
• Roofline: An Insightful Visual Performance Model for Multicore Architectures (Williams et al.)
• llama.cpp (reference implementation for optimized CPU inference)