> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Inside the Cluster > Explains llm-d's Kubernetes architecture, components, routing, sharded vLLM deployments, and how a gateway and scheduler route requests to prefill and decode pods for distributed inference. Standing up llm-d is deceptively simple: a couple of commands. That prompts two immediate questions: what do those commands create inside the cluster, and how do you actually use the running system? This article opens the hood and walks through the pieces llm-d deploys, how they map to Kubernetes objects, and how requests flow from a single endpoint to a sharded model running on many GPUs. Start at the back Every server you see in the deployment—whether labeled prefill or decode—is a vLLM instance: the same model server used in vLLM deployments. On Kubernetes, each instance runs as a pod. A set of identical pods is a Deployment, and those Deployments form your prefill and decode pools. When a model is too large for a single machine and must span multiple GPUs and nodes, you must treat the whole group of pods as one server: start it together, scale it together, and heal it together. If you already use Kubernetes you might think of a StatefulSet because it provides stable names and ordered startup. But a StatefulSet still treats each pod as an independent replica. ![A dark teal slide titled "One Server, Not Three Replicas" showing a single rounded server box that contains three smaller container-like boxes, each with a small blue shield icon. The diagram emphasizes three replicas running on one server.](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/Meet-llm-d/Inside-the-Cluster/one-server-three-replicas-diagram.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=c3998bbd87cc35062e7fa279546776d8) When a model is sharded across GPUs, its pods are pieces of a single logical server. If any shard dies, the whole server is broken and needs to be restarted — you cannot treat shards as independent replicas. Kubernetes has a newer controller/CRD designed for this exact use case: the LeaderWorkerSet. A LeaderWorkerSet declares one leader pod and its worker pods and treats that leader-plus-workers group as a single unit to start, scale, and heal. It was introduced to support sharded model deployments and is the primitive vLLM and llm-d use to distribute a large model across many machines. ![A slide diagram titled "A Component Built for This" showing a "LeaderWorkerSet" component. It depicts one leader and three worker nodes inside a container, alongside a Kubernetes icon and a "2024" label.](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/Meet-llm-d/Inside-the-Cluster/leaderworkerset-leader-workers-kubernetes-2024.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=e18aedc44f5572893fed54629badbc3c) Out front: router, gateway, scheduler At the edge of the cluster sits the router, which has two pieces: an inference gateway (the same kind of gateway you put in front of any web service) and, behind it, the llm-d scheduler — a compact decision engine that is both cache-aware and load-aware. The gateway accepts HTTP requests; the scheduler picks the right pod to handle each request. ![A dark-themed diagram titled "Out Front – The Router" showing two overlapping rounded cards labeled "Gateway" and "Scheduler" inside a dashed box. Below them are pill-shaped tags such as "Any web service", "Cache-aware", and "Load-aware".](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/Meet-llm-d/Inside-the-Cluster/out-front-router-gateway-scheduler.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=72f55ed3ffd690b501a372e05c006a1f) That yields the entire architecture: a gateway, a scheduler, and pools of vLLM pods (prefill and decode), with sharded servers grouped by LeaderWorkerSet. ![A stylized system architecture diagram titled "The Entire Architecture" showing a Gateway feeding a Scheduler that routes requests to Prefill and Decode pools of vLLM instances. Below those pools is a LeaderWorkerSet containing a leader and multiple worker nodes.](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/Meet-llm-d/Inside-the-Cluster/architecture-gateway-scheduler-vllm-pools-leaderset.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=50106f8e2d90ddbb408fdf49116c7f1a) Mapping components to Kubernetes objects If you can read a cluster, you can read this architecture. The following table maps the high-level components to the Kubernetes objects you’ll see in the cluster. | Component | Kubernetes object | Purpose | | -: | - | - | | Gateway (inference endpoint) | `Service` / Ingress | Single external entry point for all requests | | Scheduler (llm-d) | `Deployment` / `Pod` | Makes cache-aware, load-aware routing decisions | | Prefill pool | `Deployment` of vLLM pods | Reads prompts into model memory (KV cache) | | Decode pool | `Deployment` of vLLM pods | Produces token-by-token model output | | Sharded model group | `LeaderWorkerSet` CRD | Groups leader + worker pods and manages them as a single sharded server | Two config files: routing rules and the fleet Standing llm-d up is essentially two installs because you're applying two small but powerful configuration files: 1. The routing brain — a short list of plugins that define scheduling behavior. In llm-d's default config you typically see a plugin that routes requests by saved work (the KV cache) and a plugin that spreads load across replicas. ![A dark teal infographic slide titled "File 1 – The Routing Brain." It shows a "Plugins" panel with a highlighted module labeled "Route by saved work," an icon to the left, and three small rounded placeholder buttons to the right.](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/Meet-llm-d/Inside-the-Cluster/routing-brain-file1-plugins-route-module.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=e0e94741dd4fdffd595df04d7afd3bb5) 2. The fleet — the workload description. The fleet file names the model to serve and the shape of the pools behind it: how many sharded servers will read prompts (prefill) and how many will write answers (decode). Swap this file to change what you serve or how you scale. ![The image is a dark-themed infographic titled "File 2 – The Fleet" showing a 70B model and a "Read Prompts" section with three pods (pod 0, pod 1, pod 2) alongside an empty "Write Answers" panel. A purple app-like icon labeled "The Fleet" and a "Swap per workload" button appear on the left.](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/Meet-llm-d/Inside-the-Cluster/fleet-70b-read-prompts-pods.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=360b8565b7f86a094787ab955f5e116c) One honest note: the example above illustrates a GPU-sized fleet (a 70B model spanned across multiple servers) — this is the scale llm-d was built for. A small model on ordinary CPUs uses the same architecture in miniature. ![An infographic titled "The Same Shape, Two Scales" comparing a GPU-sized fleet (left) showing NVIDIA GPU icons running a 70B model with a smaller course cluster (right) of ordinary CPU icons running a small model.](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/Meet-llm-d/Inside-the-Cluster/same-shape-two-scales-gpu-cpu.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=d0edf1f925e5b7d7d2530e6e2fbb5e56) Read those two files together: routing rules + fleet. Apply them and llm-d converts that description into running pods — prefill pods, decode pods, the gateway, and the scheduler — all wired together by Kubernetes. Inspecting the running cluster A minute after applying those manifests you can inspect the cluster and see the pieces spawned from those few lines of config. Example: ```bash theme={null} $ kubectl get pods NAME READY STATUS AGE llm-d-inference-gateway-6b7d9c4f5-kq8vz 1/1 Running 62s llm-d-gaie-epp-7c9f4d8b6-x2n4p 1/1 Running 62s ms-llm-d-prefill-5f8c7b9d4-2xhtl 1/1 Running 58s ms-llm-d-prefill-5f8c7b9d4-9wkrm 1/1 Running 58s ms-llm-d-decode-84b6f5c7d-mn3qp 1/1 Running 55s ``` This is the quiet payoff of using Kubernetes primitives: the control loop you already trust manages pods, nodes, restarts, and rescheduling. Because readers (prefill) and writers (decode) are separate pools, you can scale each independently: add decode pods when answers are backing up; add prefill pods when prompts increase. You never have to babysit a single monolithic server. ![A dark-themed infographic titled "Grow Each One On Its Own" showing two sections, "Read Prompts" and "Write Answers," each containing multiple blue rounded cards labeled "vLLM" and a "+2" circle at the right. The image also includes a small © Copyright KodeKloud notice in the corner.](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/Meet-llm-d/Inside-the-Cluster/grow-each-one-vllm-read-write.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=8aec5f5baafa40d31cb21603cab6d99f) llm-d adds intelligence to Kubernetes Kubernetes keeps the servers alive; llm-d keeps them smart. The llm-d scheduler watches both the saved work (the KV cache) and the live load, routing each request to the pod that can respond best. You manage declarative configuration for the fleet and routing logic; the platform manages the running cluster. One endpoint to call When llm-d finishes bootstrapping, it hands you a single address: the inference gateway. That endpoint sits in front of the entire fleet — prefill, decode, and any sharded LeaderWorkerSet servers. You never call pods directly; call the gateway and the routing layer handles the rest. The model server used the OpenAI-compatible API originally, and that remains true. Talking to your llm-d cluster looks exactly like talking to ChatGPT: one standard HTTP request (the OpenAI chat completions API) pointed at the gateway address. Ask it to finish a sentence, and the GPUs compute the answer token-by-token, returning the result. Example request and response to the gateway: ```bash theme={null} $ curl http://inference-gateway/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","messages":[{"role":"user","content":"Serving an LLM needs"}]}' {"choices":[{"message":{"content":"GPU"}}]} ``` That single request starts a chain of coordinated work: the gateway accepts the request; the scheduler chooses the pod that already holds the relevant saved work; a prefill pod reads the prompt into memory; a decode pod produces tokens one-by-one, streaming model output from memory. If the model is sharded, an entire LeaderWorkerSet performs the computation as one logical server. All of the complexity is hidden behind the ordinary endpoint. POST /v1/chat/completions\ [http://inference-gateway/v1](http://inference-gateway/v1) Wrap-up If you already knew how to use an LLM API, you now understand what it takes to serve one: why GPUs are required for large models, why inference happens token-by-token, why the KV cache affects scheduling decisions, and why scale requires a smarter scheduling layer than basic Kubernetes primitives. From two short configuration files to a single endpoint, you can stand up a distributed inference system yourself. For more details, examples, and the project sources, visit [llm-d.ai](https://llm-d.ai). If you want to learn more about the Kubernetes primitives used here, see the Kubernetes docs: [Kubernetes Concepts](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/). > **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/ee1bfe23-198d-4b8d-a23a-e43b6730da53/lesson/36eec012-bec7-49e2-9344-9bbd1696e436)** This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.