Model Deployment Handbook

Appearance

Online Recommendation Serving

At a glance Online inference for recommender systems is one of the tightest latency-budget deployment scenarios. This guide breaks down how each model in the funnel (recall → coarse ranking → fine ranking → re-ranking) is deployed, how features are fetched, how multiple models are chained, and how the latency budget is allocated.

Online Recommendation Serving: Chaining Multiple Models Under a Latency Budget ​

One-line definition: online inference for a recommender system is a deployment engineering problem in which four stages — recall → coarse ranking → fine ranking → re-ranking — run multiple models in cascade within a hundred-millisecond latency budget. It stresses four things at once: feature engineering, model serving, cascade orchestration, and capacity planning.

Why it's worth doing: the online recommendation funnel has the tightest latency budget, the most models, and the harshest test of deployment architecture on the entire site — a 100ms end-to-end budget has to fit feature reads, vector retrieval, coarse ranking of thousands of items, fine ranking of hundreds, and re-ranking/mixing; any stage running 20ms slow blows the budget. This guide breaks down how each stage is deployed, how features are fetched online, how models are chained, and includes a deployable latency-budget allocation table. Recommendation deployment decisions lean on the traffic orchestration from the model gateway and canary releases and the budget allocation from performance optimization and capacity planning; this article is where the two come together on the front line.

1. The Recommendation Funnel and Its Latency Budget ​

text
User request
   │
   ▼
┌───────────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│    Recall     │──▶│    Coarse     │──▶│     Fine      │──▶│    Re-rank      │──▶ Results
│   10M → 1K    │   │  1K → 100s    │   │  100s → 10s   │   │ dedup/diversity │
└───────────────┘   └───────────────┘   └───────────────┘   └─────────────────┘
 vector search       two-tower /        deep ranking        rules + small
                     small models       models              models
StageCandidate volumeTypical modelsLatency budget
Recall10M → 1–2KVector retrieval (Faiss/ANN) + two-tower model15–20ms
Coarse ranking1K → 100–300Two-tower / small DNN (trimmed feature set)10–15ms
Fine rankingHundreds → dozensDeep CTR model (full feature set)30–50ms
Re-rankingDozens → what gets shownRules + small models (diversity, freshness)5–10ms
Feature/context reads—Feature Store / KV15–25ms
Total≤ 100ms

Verdict first: the allocation principle is "the earlier the stage, the cheaper the compute" — recall handles many candidates with light models, while fine ranking handles few candidates with heavy models; save the expensive computation for the few hundred items worth fine ranking. Tune the exact numbers to your business, but always keep 20% of the total budget in reserve for network jitter and unforeseeable overhead.

2. Putting Feature Engineering Online ​

Fine-ranking models live on features, and feature inconsistency is the number-one source of incidents in online recommendation. Three practices:

  1. Split features into real-time and offline paths: offline features (long-term user profiles, item attributes) are pushed into the Feature Store daily by batch inference pipelines; real-time features (recent click sequences, context) are computed online and written to Redis-like stores under TTL control;
  2. Feature consistency: training must use features as of the request's point in time, and online serving must do the same — no peeking at the future. The most common incident is "training used features as of time T, online serves features as of T+1," and the model falls apart. Consistency governance is covered under the feature leakage entry in common pitfalls and antipatterns;
  3. Feature store selection: read-heavy with millisecond latency → KV stores such as Redis/Faiss; need replay and lineage → a Feature Store (Feast, Alibaba Cloud FeatureStore, etc.). Online is read-only; offline writes in batches — avoid coupling the online path by writing to the online store from the serving side.

3. Serving the Fine-Ranking Model ​

Fine ranking is the latency hog. Three key deployment moves:

1. Embedding lookup and input assembly: features split in two — numeric features go straight into the model, while sparse ID features first hit an embedding lookup (KV/vector store) and are then assembled into tensors. Should assembly happen inside the model service or in an outer layer? Verdict: put assembly inside the service and pass business semantics in the input schema (user_id, item_ids); the model service wraps "lookup → assemble → forward → post-process" into a single predict call — the same encapsulation principle as the FastAPI walkthrough, but latency-sensitive, so you'll typically use gRPC instead of HTTP (see serving and inference APIs).

2. Batched inference: fine ranking receives hundreds of candidates at once — always score them all in one forward pass, never one at a time. A deep CTR model stacks item_ids along the batch dimension [N, feature_dim] and gets N scores from a single forward. This is the "batching" play from performance optimization and capacity planning; for many models or higher throughput, add Triton dynamic batching.

python
# Core fine-ranking service logic (illustrative)
import torch

@torch.inference_mode()
def rank(user_vector, item_ids, item_emb_cache):
    # item_emb_cache: [N, d] embedding matrix fetched from the KV store
    item_emb = torch.from_numpy(item_emb_cache)          # [N, d]
    user = user_vector.unsqueeze(0).expand(len(item_ids), -1)  # [N, d]
    x = torch.cat([user, item_emb, cross_features], dim=1)     # assemble the input
    scores = ctr_model(x).squeeze(-1)                    # one forward pass yields N scores
    return scores

3. Model size and memory: fine-ranking models are large and GPU memory is expensive; the usual cost levers are quantization (FP16/INT8, see quantization) and tiering the embedding table by heat (hot IDs fully resident in memory, cold IDs fetched from external storage).

The Fine-Ranking Contract: Assemble Once, Forward Once ​

The fine-ranking service's external schema uses business semantics rather than raw tensors; the model service handles "lookup → assemble → forward" internally:

json
// Request: one user + N candidates
{
  "request_id": "req_8f21",
  "user": { "user_id": "u_10086", "scene": "home_feed" },
  "items": ["i_101", "i_102", "i_103"],
  "context": { "hour": 21, "device": "android" }
}
// Response: scores matching the request items one-to-one
{
  "request_id": "req_8f21",
  "scores": [0.91, 0.82, 0.76]
}

The contract carries a request_id so that the whole funnel can be reconciled — the recall/coarse/fine stage logs all carry the same id, so when you ask "why didn't this candidate reach fine ranking," you can stitch the trail together directly. This is the standard use of traceability from monitoring and observability in a recommendation context.

The Latency Ledger for Feature Assembly ​

Within a 40ms fine-ranking budget, feature fetching is an easily underestimated hidden cost. A reference breakdown (all KV hits, single-node Redis):

OperationTimeNotes
Bulk user-profile fetch (1 batched GET)~0.5–1msBatched feature-store reads
Bulk item-feature fetch for N candidates~1–2msN=300, batched rather than per-item
Embedding lookup~2–5msVector store / in-memory KV
Input assembly (concat/numpy)~1msDon't assemble field by field in Python
Model forward (batch=300)~10–25msThe fine-ranking model itself

Verdict first: feature fetching normally takes only 10–20% of the fine-ranking budget, but fetching per item inflates it 10× — batch your reads at every stage; never do N round trips for N items. This is priority one for funnel optimization: fix the naive per-item fetching before you touch the model.

4. Chaining Models: Cascades, Timeouts, and Degradation ​

Four stages, four services; the heart of the chain is timeouts and degradation:

RiskMitigation
Slow upstream drags down downstreamIndependent per-stage timeouts (e.g. recall 20ms); on timeout, serve the previous batch's cached results
Recall is downDegrade: serve only the hot pool / a fixed homepage pool to preserve availability
Fine ranking is slowDegrade: skip fine ranking and send coarse-ranking results straight to re-ranking
A single point saturatesIndependent scaling per stage + rate limiting; see the model gateway and canary releases

Three ways to implement it; the verdict: complex business logic (multiple branches, personalized degradation) → application-layer orchestration (gateway or in-service calls); a fixed, simple funnel → Triton ensembles or a KServe inference graph. Timeout references: 100ms end-to-end, with an independent timeout reserved per stage (recall 20ms, coarse ranking 15ms, fine ranking 40ms, re-ranking 10ms), plus a Hystrix/Sentinel-style circuit breaker protecting downstream — trip fast when a stage fails repeatedly, before the cascade melts down.

5. Caching and Hot-Key Handling ​

Two workhorses of online recommendation latency optimization:

  1. Result caching: the same user + same scene repeats requests within a short window and hits the cache directly. Key = user_id + scene + model_version. The cache key must include the model version — otherwise the model updates and you keep serving old results, the same trap as the "cache pollution" problem in the model gateway and canary releases;
  2. Hot traffic: head users contribute most of the traffic. Countermeasures: pre-warm head users' recommendations into the cache (computed in the previous full run) and pin hot items' embeddings in memory ahead of time. Hot-key treatment is discussed under "hot spots and caching" in deployment architecture patterns.

6. Architecture and Tooling Choices ​

text
Client ─▶ Recommendation gateway (routing / rate limiting / canary / A-B)
              │
              ▼
   Recommendation orchestration service (cascade + timeouts + degradation)
    │               │               │
    ▼               ▼               ▼
   Recall        Coarse rank     Fine rank ──▶ Re-rank ──▶ Output
   (Faiss +      (small DNN)     (deep CTR)    (rules/small
    two-tower)                                  models)
    │               │               │
    ▼               ▼               ▼
   Feature Store (KV/Redis + offline profile pipeline) + Vector store (Milvus/Faiss)
ComponentOptionsOne-line rationale
Feature storeRedis / FeastMillisecond reads, batch writes, TTL
Vector retrievalFaiss / MilvusThe recall battleground: ANN search
Model servingFastAPI (small models) / Triton (large, multiple models)Pick by latency and throughput needs
Recommendation gatewayEnvoy / in-houseRouting, rate limiting, canary, A/B
OrchestrationIn-house orchestration service / Triton ensembleCascade, timeouts, degradation

For the practical method on latency budgets and capacity planning, see performance optimization and capacity planning; rolling out a new recommendation model goes through the full model gateway and canary release flow (the gateway here is its entry point); keeping model versions and features aligned is the job of the MLOps deployment pipeline.

Common Pitfalls and Troubleshooting ​

PitfallSymptomFix
Feature leakageOnline performance far worse than offlineTime-align training/online features; see Section 2
Missing cascade timeoutsOne slow stage → the whole funnel times outIndependent per-stage timeouts + circuit breakers
Per-item fine rankingFine ranking takes 200msBatch the candidates into one forward pass
Cache without a versionResults don't change after a model updateAdd model_version to the cache key
Brute-force scan of the recall poolRecall blows the budgetUse ANN (HNSW/IVF); never brute-force the full pool
Head users saturate the serviceHot-user requests drag everything downPre-warm hot results into the cache
No degradation strategyFine ranking dies and the site's recommendations go blankPrebuild a degradation chain (skip fine ranking / hot-pool fallback)

Further Reading ​

References ​