Anatomy of an Inference System: A Seven-Layer Panorama

A production-grade inference system isn't "a model plus a Flask app" — it's a systems-engineering effort with a seven-layer structure: request entry, the serving layer, the inference engine, and the resource layer, plus three cross-cutting support planes: data and versioning, observability, and governance. This page establishes the framework with one panorama diagram, then dissects each layer's responsibilities, tooling choices, and common failures, and finally ties all seven together by walking through the life of a single request.

In one sentence: an inference system is the sum of every component that makes "request → prediction" happen stably, cheaply, and observably.

1. The Overall Architecture: A Seven-Layer Panorama

                     Clients (app / web / upstream services)
                                      │
                                      ▼
┌───────────────────────────────────────────────────────────────────┐
│ ① Request entry layer                                             │
│   Gateway/load balancing: routing, auth, rate limiting, timeouts  │
│   (Nginx / Envoy / API gateway)                                   │
├───────────────────────────────────────────────────────────────────┤
│ ② Serving layer                                                   │
│   API (REST/gRPC) → preprocessing (tokenize/normalize) →          │
│   inference scheduling → postprocessing                           │
│   FastAPI / Triton / KServe / vLLM                                │
├───────────────────────────────────────────────────────────────────┤
│ ③ Inference engine layer                                          │
│   Graph optimization, quantized kernels, batching executor        │
│   (ONNX Runtime / TensorRT / vLLM)                                │
├───────────────────────────────────────────────────────────────────┤
│ ④ Resource layer                                                  │
│   GPU / CPU / memory / network; K8s nodes and scheduling          │
├───────────────────────────────────────────────────────────────────┤
│ ⑤ Data & versioning layer (cross-cutting)                         │
│   Model registry (MLflow), feature versions, tokenizer/vocab      │
├───────────────────────────────────────────────────────────────────┤
│ ⑥ Observability layer (cross-cutting)                             │
│   Metrics (Prometheus), logs, tracing, drift detection            │
├───────────────────────────────────────────────────────────────────┤
│ ⑦ Governance layer (cross-cutting)                                │
│   Canaries, rollbacks, capacity planning, security and quotas     │
└───────────────────────────────────────────────────────────────────┘

Suggested reading order: first understand the vertical spine ①→④ (how a request becomes a prediction), then the three cross-cutting planes ⑤⑥⑦ (they do no computation, but they determine whether the system survives long-term).

A mnemonic for the spine

Entry controls admission, serving orchestrates, the engine computes, resources run it — the four layers of the spine in one line.

2. Layer by Layer: Responsibilities, Tooling, and Failures

1. Request entry layer: controlling "who gets in"

• Responsibilities: load balancing, routing, authentication, rate limiting, timeout/circuit breaking, TLS termination.
• Typical choices: Nginx, Envoy, Kong; Kubernetes Ingress; cloud API gateways.
• Common failures: failed health checks pulling all traffic out of rotation; rate limits set too loose, letting a request flood punch straight through to the serving layer.
• Where to look: gateway access logs and latency distributions, to distinguish "never got in" from "got in but slow."

2. Serving layer: turning "a request into one inference"

• Responsibilities: parsing requests, input validation, preprocessing (image resize, text tokenization), assembling the inference request, postprocessing (softmax to scores, decoding), building the response. Dynamic batching policy is orchestrated here too.
• Typical choices: FastAPI, Triton, vLLM, KServe; for LLMs, vLLM/Triton typically handle serving and scheduling directly.
• Common failures: request schema mismatches (missing fields, wrong types); preprocessing slower than inference; poor batching policy causing tail latency (head-of-line blocking).
• Where to look: instrument preprocessing / inference / postprocessing durations separately — see Putting Observability into Practice.

Two shapes of the serving layer

Traditional CV/NLP models often follow the "write your own FastAPI that calls an engine" pattern; LLMs generally use all-in-one frameworks like vLLM, where the serving layer and the engine layer merge. Different shapes, same orchestration responsibility.

3. Inference engine layer: making "computation fast"

• Responsibilities: optimizing the model graph into an efficient execution plan — operator fusion, quantized kernels (INT8/FP16), memory reuse, batched kernel execution.
• Typical choices: ONNX Runtime (cross-platform), TensorRT (peak performance on NVIDIA GPUs), vLLM (LLM scheduling + PagedAttention), OpenVINO (Intel CPUs). Comparison: Choosing Frameworks and Platforms.
• Common failures: unsupported operators falling back to slow paths; memory fragmentation/OOM; accuracy drift after quantization.
• Where to look: engine profiling tools (NVIDIA Nsight, ONNX Runtime profiler) to get operator-level timings — see Performance Optimization and Capacity Planning.

4. Resource layer: ensuring "there's a machine to run on"

• Responsibilities: provisioning and scheduling GPU/CPU compute, VRAM, memory, and network bandwidth; node scaling.
• Typical choices: Kubernetes node pools, GPU-sharing schedulers (MIG, time slicing), serverless autoscaling.
• Common failures: a neighbor pod exhausting GPU memory; node OOM killing pods; slow cold starts while pulling images.
• Where to look: K8s events and node metrics, to distinguish "not enough resources" from "code wasting resources."

5. Data & versioning layer (cross-cutting): controlling "which model version is in use"

• Responsibilities: model registry entries, image tags, feature versions, tokenizer/vocab alignment. This is the layer where model systems most often fail silently: the model changed, the features didn't.
• Typical choices: MLflow Model Registry, Hugging Face Hub, in-house version tables.
• Common failures: production inference using feature definitions inconsistent with training; a new model shipping while the cache still holds old features.
• Where to look: compare model/feature version commits against go-live timestamps. Full workflow in MLOps Deployment Pipelines.

6. Observability layer (cross-cutting): catching "is it quietly degrading?"

• Responsibilities: metrics (latency, throughput, error rate, GPU utilization), logs, tracing, drift detection, alerting.
• Typical choices: Prometheus + Grafana, OpenTelemetry, ELK/Loki, drift-detection scripts.
• Common failures: metric sampling too coarse to catch occasional spikes; no alerting, so drift goes unnoticed for weeks.
• Where to look: start from SLO metrics, then drill into logs. Metric system design in Monitoring and Observability.

7. Governance layer (cross-cutting): managing "change and risk"

• Responsibilities: canary releases, rollbacks, capacity planning, quotas, security and compliance (model permissions, data masking), model audit trails.
• Typical choices: K8s rolling updates, Argo Rollouts, gateway-based canaries (A/B, canary), model audit logs.
• Common failures: a full release going wrong with nothing but a wholesale rollback available; uneven traffic splitting in canaries skewing experiment conclusions.
• Where to look: canary practice in Model Gateway and Canary Releases; security baseline in Security, Privacy, and Compliance.

3. Walking Through the Life of a Request

Tying the seven layers together — how an HTTP request becomes a prediction:

1. App sends POST /predict (JSON: {feature_a: 0.7, text: "..."})
     ↓ ① Request entry layer
2. Gateway: auth passes → rate-limit counter → route to serving pod
     ↓ ② Serving layer
3. API parses the request and validates the schema; invalid → 422
4. Preprocessing: tokenize text / resize + normalize image
5. Assemble the batch (dynamic batching merges 4 requests into one)
     ↓ ③ Inference engine layer
6. Engine executes the graph: fused forward pass, INT8 kernels, memory reuse
7. Raw outputs returned (logits / embeddings)
     ↓ ② Serving layer
8. Postprocessing: softmax → top-k scores / detokenize text
9. Build response JSON (with latency headers) → return to app
     ↓ ⑥ Observability layer (recorded alongside, end to end)
10. Latency histograms, batch size, error codes reported to Prometheus
     ↓ ⑤ Data & versioning layer (verified alongside)
11. Model and feature versions used for this request recorded for audit and replay

Key observation: ①→④ determine whether it's fast; ⑤→⑦ determine whether it's stable. When an interviewer asks "which layers does a request pass through," answer along the spine ①→④, then add the three cross-cutting planes — that's a full-mark structure.

4. Contracts Between Layers: Stability Comes from Protocols, Not Goodwill

Every layer boundary needs an explicit contract — otherwise any single layer's upgrade can shatter its neighbors:

Contract	What it covers	What breaking it looks like
Input/output schema	Field names, types, value ranges, defaulting policy (JSON Schema / gRPC proto)	Old and new upstream versions mixed, parsing errors
Batching protocol	Single vs. batch, batch size limits, timeout semantics	Batching chaos, runaway tail latency
Error semantics	4xx/5xx split, error codes, retry safety (idempotency)	Retries amplifying the failure (retry storms)
Version alignment	Model version ↔ feature version ↔ engine version ↔ image tag	Silent accuracy degradation, hard to localize

The most neglected contract: retries

Gateways commonly retry 5xx responses. If the inference endpoint isn't idempotent (say, each call charges a fee), one timeout can trigger three executions. The contract must state which errors are retryable, how many times, and with what backoff.

Full design of serving-layer contracts: Model Serving and Inference APIs.

5. Where Failures Show Up: From Symptom to Layer

The same symptom can originate in different layers. Step one of troubleshooting is mapping the symptom back to a layer:

Symptom	Most likely layer	How to verify
All requests time out	① Entry layer (gateway config / upstream unreachable)	Gateway logs + health-check status
Occasional P99 spikes	② Serving layer (batch queueing)	Per-stage timing breakdown
Steadily getting slower	③ Engine layer (fallback to slow kernels)	Engine profiler
OOM / out of VRAM	④ Resource layer	Node metrics + engine memory logs
Correct results, occasionally wrong	⑤ Versioning layer (feature drift)	Version comparison + sample replay
Quiet quality decay in production	⑥ Observability layer (drift unalerted)	Drift metric trends
Problems after a new release	⑦ Governance layer (canary didn't catch it)	Canary logs + rollback

A troubleshooting mantra

Split into stages before drilling down: first break "request in to response out" into four instrumented segments — gateway, serving, engine, resources — then dive into whichever is slow. Don't start by suspecting engine kernels; the odds are the problem sits in the serving or resource layer. The full methodology: Common Pitfalls and Anti-Patterns.

6. Using the Seven Layers: Three Roles, Three Uses

• Deployment engineers: treat ①→④ as the "performance and stability map" and ⑥⑦ as the "daily operations panel" — and always optimize from a measured bottleneck.
• ML/algorithm engineers: understand how critical ⑤ is — when "my model worked in training but doesn't in production," half the answer lies in feature and version alignment.
• Interviewees: the architecture diagram and request walkthrough here are a repeatable "system design" framework — practice with the interview questions until you can draw it from memory.

Further Reading

• What Is Model Deployment? — the definitions and four elements behind the seven layers
• Model Serving and Inference APIs — batching and API design for the serving layer (②)
• Performance Optimization and Capacity Planning — optimization methodology for the engine (③) and resource (④) layers
• Monitoring and Observability — the metric system for the observability layer (⑥)
• MLOps Deployment Pipelines — process support for the data & versioning layer (⑤)
• Model Gateway and Canary Releases — a complete hands-on for the governance layer (⑦)

References

• The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction (IEEE 2017)
• Uber – Michelangelo: machine learning platform architecture in practice
• Netflix – the evolution of ML infrastructure
• Martin Fowler – Continuous Delivery and deployment pipelines (a deployment automation reference)
• Prometheus documentation (the observability layer's metrics tool)