Explains llm-d’s Kubernetes architecture, components, routing, sharded vLLM deployments, and how a gateway and scheduler route requests to prefill and decode pods for distributed inference.
Standing up llm-d is deceptively simple: a couple of commands. That prompts two immediate questions: what do those commands create inside the cluster, and how do you actually use the running system? This article opens the hood and walks through the pieces llm-d deploys, how they map to Kubernetes objects, and how requests flow from a single endpoint to a sharded model running on many GPUs.Start at the backEvery server you see in the deployment—whether labeled prefill or decode—is a vLLM instance: the same model server used in vLLM deployments. On Kubernetes, each instance runs as a pod. A set of identical pods is a Deployment, and those Deployments form your prefill and decode pools.When a model is too large for a single machine and must span multiple GPUs and nodes, you must treat the whole group of pods as one server: start it together, scale it together, and heal it together. If you already use Kubernetes you might think of a StatefulSet because it provides stable names and ordered startup. But a StatefulSet still treats each pod as an independent replica.
When a model is sharded across GPUs, its pods are pieces of a single logical server. If any shard dies, the whole server is broken and needs to be restarted — you cannot treat shards as independent replicas.
Kubernetes has a newer controller/CRD designed for this exact use case: the LeaderWorkerSet. A LeaderWorkerSet declares one leader pod and its worker pods and treats that leader-plus-workers group as a single unit to start, scale, and heal. It was introduced to support sharded model deployments and is the primitive vLLM and llm-d use to distribute a large model across many machines.
Out front: router, gateway, schedulerAt the edge of the cluster sits the router, which has two pieces: an inference gateway (the same kind of gateway you put in front of any web service) and, behind it, the llm-d scheduler — a compact decision engine that is both cache-aware and load-aware. The gateway accepts HTTP requests; the scheduler picks the right pod to handle each request.
That yields the entire architecture: a gateway, a scheduler, and pools of vLLM pods (prefill and decode), with sharded servers grouped by LeaderWorkerSet.
Mapping components to Kubernetes objectsIf you can read a cluster, you can read this architecture. The following table maps the high-level components to the Kubernetes objects you’ll see in the cluster.
Component
Kubernetes object
Purpose
Gateway (inference endpoint)
Service / Ingress
Single external entry point for all requests
Scheduler (llm-d)
Deployment / Pod
Makes cache-aware, load-aware routing decisions
Prefill pool
Deployment of vLLM pods
Reads prompts into model memory (KV cache)
Decode pool
Deployment of vLLM pods
Produces token-by-token model output
Sharded model group
LeaderWorkerSet CRD
Groups leader + worker pods and manages them as a single sharded server
Two config files: routing rules and the fleetStanding llm-d up is essentially two installs because you’re applying two small but powerful configuration files:
The routing brain — a short list of plugins that define scheduling behavior. In llm-d’s default config you typically see a plugin that routes requests by saved work (the KV cache) and a plugin that spreads load across replicas.
The fleet — the workload description. The fleet file names the model to serve and the shape of the pools behind it: how many sharded servers will read prompts (prefill) and how many will write answers (decode). Swap this file to change what you serve or how you scale.
One honest note: the example above illustrates a GPU-sized fleet (a 70B model spanned across multiple servers) — this is the scale llm-d was built for. A small model on ordinary CPUs uses the same architecture in miniature.
Read those two files together: routing rules + fleet. Apply them and llm-d converts that description into running pods — prefill pods, decode pods, the gateway, and the scheduler — all wired together by Kubernetes.Inspecting the running clusterA minute after applying those manifests you can inspect the cluster and see the pieces spawned from those few lines of config. Example:
This is the quiet payoff of using Kubernetes primitives: the control loop you already trust manages pods, nodes, restarts, and rescheduling. Because readers (prefill) and writers (decode) are separate pools, you can scale each independently: add decode pods when answers are backing up; add prefill pods when prompts increase. You never have to babysit a single monolithic server.
llm-d adds intelligence to KubernetesKubernetes keeps the servers alive; llm-d keeps them smart. The llm-d scheduler watches both the saved work (the KV cache) and the live load, routing each request to the pod that can respond best. You manage declarative configuration for the fleet and routing logic; the platform manages the running cluster.One endpoint to callWhen llm-d finishes bootstrapping, it hands you a single address: the inference gateway. That endpoint sits in front of the entire fleet — prefill, decode, and any sharded LeaderWorkerSet servers. You never call pods directly; call the gateway and the routing layer handles the rest.The model server used the OpenAI-compatible API originally, and that remains true. Talking to your llm-d cluster looks exactly like talking to ChatGPT: one standard HTTP request (the OpenAI chat completions API) pointed at the gateway address. Ask it to finish a sentence, and the GPUs compute the answer token-by-token, returning the result.Example request and response to the gateway:
That single request starts a chain of coordinated work: the gateway accepts the request; the scheduler chooses the pod that already holds the relevant saved work; a prefill pod reads the prompt into memory; a decode pod produces tokens one-by-one, streaming model output from memory. If the model is sharded, an entire LeaderWorkerSet performs the computation as one logical server. All of the complexity is hidden behind the ordinary endpoint.POST /v1/chat/completions http://inference-gateway/v1Wrap-upIf you already knew how to use an LLM API, you now understand what it takes to serve one: why GPUs are required for large models, why inference happens token-by-token, why the KV cache affects scheduling decisions, and why scale requires a smarter scheduling layer than basic Kubernetes primitives. From two short configuration files to a single endpoint, you can stand up a distributed inference system yourself.For more details, examples, and the project sources, visit llm-d.ai.
If you want to learn more about the Kubernetes primitives used here, see the Kubernetes docs: Kubernetes Concepts.
