Inside the Cluster

This document explains how llm-d runs inside a Kubernetes cluster: how the Helm releases map to Kubernetes objects, how model servers (vLLM) are deployed and scaled, and how routing and sharded models are coordinated. Two Helm releases set up a model deployment:

A Helm release that installs the llm-d router plus the small control-plane components (inference gateway + scheduler).
A Helm release that installs the model service (the vLLM-based servers).

Example Helm commands:

$ helm install router llm-d/router \
  -f router.values.yaml

$ helm install llama llm-d/modelservice \
  -f llama-70b.values.yaml

Start from a Kubernetes cluster with available worker nodes. Initially there are no model pods running. When you deploy a model, each model server is a vLLM instance (see https://github.com/vllm-project/vllm). In llm-d, each vLLM process runs inside a pod scheduled onto nodes with available GPU/CPU resources. A group of identical pods serving the same role is represented by a Deployment. llm-d uses two conceptual pools for each model deployment:

Prefill pool: pods that read prompts and build/restore model execution state (prepare cache).
Decode pool: pods that produce tokens and stream outputs.

Prefill pods and decode pods are scaled independently so you can tune read-heavy vs. decode-heavy workloads. So what about very large models that are too big for one machine? You might consider StatefulSets because they provide stable pod identities and an ordinal startup order. That pattern is useful for databases and other services with independent replicas.

StatefulSets model each pod as an independent replica. For a sharded LLM, shards are not independent — they combine to form one logical server. Losing a single shard makes the whole model unusable, so scale and healing must treat the shard group as a unit.

llm-d addresses this by grouping a leader pod with worker pods into a single logical unit (commonly referred to as a LeaderWorkerSet). That grouping is scaled and healed together, so a sharded model behaves like a single server composed of coordinated parts.

All components — prefill pods, decode pods, gateway, scheduler, and any sharded groups — are expressed to Kubernetes via manifests generated by Helm. The values.yaml you pass to Helm declares the model artifacts and the replica counts for each pool. Updating those values and reapplying the Helm release changes the deployed fleet topology. Example values.yaml fragment (model artifacts and pool sizes):

# the fleet
modelArtifacts:
  uri: hf://meta-llama/llama-3-70B

prefill:
  replicas: 4  # pods that read prompts

decode:
  replicas: 1  # pods that write tokens

After you install the Helm releases, llm-d will create the prefill and decode pods, deploy the inference gateway and scheduler, and wire the pieces together. Use kubectl to inspect the running pods:

$ kubectl get pods
NAME                      STATUS
llama-70b-prefill-0       Running
llama-70b-prefill-1       Running
llama-70b-prefill-2       Running
llama-70b-prefill-3       Running
llama-70b-decode-0        Running
inference-gateway-...     Running

Because this runs on Kubernetes, you get standard platform guarantees: crashed pods are restarted, pods on failed nodes are rescheduled, and you can scale prefill and decode pools independently. llm-d handles model-aware routing, caching, and request orchestration while Kubernetes handles liveness, scheduling, and rescheduling.

An illustrated diagram titled "Kubernetes Keeps Them Alive" showing a cluster with a gateway, scheduler, and multiple prefill and decode pods handling incoming requests. There's also a small circular presenter video overlay in the bottom-right corner.

llm-d exposes a single public endpoint via the inference gateway. Clients send all requests to that gateway — you never send requests directly to individual pods. The scheduler routes requests to the appropriate prefill and decode pods based on cache locality and current load.

Get the gateway address with kubectl:

$ kubectl get gateway
NAME                 ADDRESS
inference-gateway    203.0.113.10

When a client request hits the gateway, the scheduler selects a pod (or shard group) that already holds the needed cached state when possible. The prefill pod reads the prompt and prepares model state; the decode pod generates tokens and streams them, token-by-token, back through the gateway to the client.

For massive sharded models, token generation is distributed across GPUs on multiple machines. The LeaderWorkerSet coordinates those shards so they behave as a single logical model server while preserving efficient cross-machine computation. Component summary

Component	Role	Example / Notes
Inference gateway	Single public endpoint for clients	`inference-gateway` with external IP
Scheduler	Routes requests to pods based on cache/load	Coordinates prefill→decode flow
Prefill pods	Read prompts and restore model state	Scaled independently (`prefill.replicas`)
Decode pods	Generate tokens and stream outputs	Scaled independently (`decode.replicas`)
LeaderWorkerSet	Group of pods representing a sharded model	Scaled and healed as a unit (not a StatefulSet)
vLLM	Inference server runtime inside pods	https://github.com/vllm-project/vllm

Links and References

llm-d project: https://llm-d.ai
vLLM (inference server runtime): https://github.com/vllm-project/vllm
Kubernetes documentation: https://kubernetes.io/docs/
Helm documentation: https://helm.sh/docs/

An Introduction to AI Infrastructure

From a GPU to a Model Server

What Happens When You Interact with LLM

Why Serving Gets Hard

Meet llm-d

Inside the Cluster

Watch Video