Skip to main content
A serving fleet never stands still. Whether the model server ships a new binary, the base model is upgraded, or a team needs their fine‑tuned flavor deployed by Friday, changes must happen under live traffic — there is no maintenance window when users are mid‑conversation.
A dark-slide graphic titled "The Fleet Never Stands Still" showing a three-step flow: "New Server Version" → "Model Upgrade" → "Fine‑Tuned Flavor." Each step appears in a rounded colored box with simple icons (server, robot, wrench).
This article explains the three rollout strategies taught in the llm‑d: A Deep Dive course and shows which approach our standalone cluster will use for live updates. We’ll preserve the order of the technical diagrams and keep the practical commands and examples where they belong. Why model serving rollouts are different
  • Pods are heavy: model weights are gigabytes, and loading them takes minutes. You can’t churn instances casually.
  • Pods are effectively stateful: their in‑memory context contains user conversations; restarting a pod can force a conversation to be re-read.
  • Requests are long: individual decodes can take seconds, so restarting a pod can interrupt in‑flight answers.
A dark-themed slide titled "Respect the Problem" with horizontal gradient progress bars comparing "web app" and "vLLM" for the categories "Weights" and "Time to Ready." The bars show different lengths to indicate relative values.
Readiness probes are the core tool for zero‑downtime: they do more than check that a process started. A model‑server readiness probe must assert that the model and its dependencies are fully loaded and can serve inferences reliably. That allows Kubernetes to keep a new pod NotReady while it finishes the slow initial load.
A slide titled "The Pods Are Stateful" showing a user icon pointing to a vLLM container that contains three small pod windows, with an orange X marker at the container's corner.
A dark-themed diagram titled "The Requests Are Long" showing a vLLM block on the left, a long progress bar with spaced green dots along a dotted line, and a user icon on the right. It visually represents a long request/processing pipeline from vLLM to the user.
Kubernetes’ rolling update plus readiness probes
  • Kubernetes replaces pods one (or a small batch) at a time by default.
  • A readiness probe that requires the model to be loaded prevents traffic from reaching pods that are not yet able to serve.
  • New pods can spend minutes loading weights while staying out of the service’s endpoint pool.
Here’s how Kubernetes shows a NotReady pod during the load phase:
Readiness probes should reflect the real serving state (model loaded, dependencies initialized). Do not equate readiness with merely “process started”; readiness must mean “can accept and complete inferences without dropping requests.”
The slow first minutes are therefore fenced off by design: the router (service or load‑balancer) only routes to Ready endpoints. The usual sequence during a rolling update is: bring up a new pod, wait for it to load and become Ready, add it to the pool, drain an old pod, and repeat.
A dark teal presentation slide titled "Fenced Off by Design" showing three stacked rounded boxes labeled "vLLM" on the right, the bottom box containing a progress bar. An arrow from that box points to a small clock icon.
A dark-themed diagram titled "One Pod at a Time" shows a "Decode Pool" containing several rectangular vLLM pod cards labeled v1 and v2, with one pod displaying a progress bar and a green check. The lower-left corner shows a copyright notice for KodeKloud.
Rolling update is the default strategy and the one our standalone cluster uses. It shrinks and swells the pool by one (or a configured small number) without blinking the service. Blue‑green for controlled exposure Sometimes “one‑at‑a‑time” is still too coarse. If you want to observe a new base model under real traffic gradually, stand up a parallel pool (green) beside the production pool (blue) and split traffic at the front door: 1% → 10% → 100% while you monitor metrics. Blue remains an instant rollback during the test.
A diagram titled "Splitting at the Front Door" shows a home/front-door icon routing traffic into two lanes labeled Blue and Green, each containing vLLM service boxes. The Green lane is annotated with "10%".
Blue‑green rollouts require gateway-level weighted traffic splitting. Standalone mode (without a central gateway) makes exact small‑percentage splits hard to achieve. Use a Gateway API or dedicated edge gateway when you need precise canary percentages.
Honest note for our cluster: weighted front‑door splitting belongs to gateway machinery (e.g., a Gateway API with an NGINX Fabric Gateway). Standalone mode deliberately runs without that centralized gateway, so precise 1% vs 99% weight control is not reliable from inside an Envoy instance in a pod.
A slide titled "A Job for the Fuller Stack" showing two deployment options: Standalone Mode with an "L7 proxy" icon on the left, and Gateway Mode on the right with "Gateway" and "Scheduler" icons connected by an arrow. The diagram sits on a dark teal background with dashed bounding boxes around each mode.
Adapter swaps (LoRA) for quick, low‑risk updates Many day‑to‑day changes are small: teams ship a new fine‑tuned flavor rather than swapping the giant base model. LoRA (Low‑Rank Adaptation) keeps adapters tiny — a small set of parameters that load alongside a frozen base model and steer its outputs. Because adapters are orders of magnitude smaller than the base, they can be swapped with minimal disruption. Example adapter weight snippets:
A dark presentation slide titled "Low-Rank Adaptation" showing two horizontal progress-style bars and three rounded app-style icons, including a large pink robot icon labeled "Giant." The top bar is pink and nearly full while the lower bar is dark with a small blue knob.
Because the base model stays warm in GPU memory, swapping adapters is fast and low risk. Different teams can independently deploy adapters (support on Tuesday, legal on Thursday) without reloading gigabytes of base weights. This makes adapter swaps an efficient default for frequent, small updates. Comparing the three strategies
A presentation slide titled "Three Strategies, One Theme" showing three rounded boxes labeled "Rolling Update" (highlighted), "Blue‑Green", and "Adapter Swap" on a dark background.
One theme across all three strategies: keep the fleet answering while you change it. Our standalone cluster already uses the rolling‑update pattern with readiness probes, and that’s the demo we’ll run next: a rolling update under live traffic while we count any failures. References and further reading Let’s run a rolling update under live traffic and count the failures.

Watch Video