> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Demo Read the PD well lit Path Against Our Live Cluster > Explains implementing software-based prefill and decode disaggregation for LLM serving on Kubernetes using role labels, sidecars, EPP scheduling, and NIXL KV transfers. Earlier we discussed why the largest fleets separate prefill from decode: the two phases stress hardware in opposite ways. A big read (prefill) moves large amounts of prompt context through memory at once, while streaming (decode) emits tokens one-by-one. Splitting those roles into dedicated pools lets you tune prefill and decode independently — in practice this can yield significantly higher tokens-per-second on the same hardware. Long prompts, mixed traffic, and lock-step expert groups are workloads where the split delivers the most benefit. Splitting prefill and decode typically improves throughput because prefill optimizes bulk memory access and cache warm-up, while decode optimizes per-token latency and streaming behavior. Why not run a full hardware-backed split in this demo? A true, fully disaggregated P/D system depends on extremely fast pod-to-pod links and memory-visible transfers across hosts — hardware our small CPU cluster doesn't provide. Instead of faking a hardware topology, we inspect what the cluster actually runs, read the real P/D manifests, and show how close a control-plane + software P/D path can get to a full split. The code, labels, sidecars, and scheduler wiring are all present; we simply avoid claiming hardware-level transfer performance we don't have. Start by listing pod labels in the llm-d namespace: ```bash theme={null} $ kubectl get pods -n llm-d --show-labels ``` Look at the label on the model pods: `llm-d.ai/role=decode`. Every model pod currently uses that value — a single fleet handling both prompt-reading and streaming. ```bash theme={null} $ kubectl get pods -n llm-d --show-labels NAME READY STATUS RESTARTS AGE optimized-baseline-cpu-vllm-decode-5c77c88766-4hmpb 1/1 Running 0 5d16h llm-d.ai/accelerator-variant=cpu,llm-d.ai/accelerator-vendor=intel, llm-d.ai/guide=optimized-baseline,llm-d.ai/model=Qwen2.5-0.5B-Instruct, llm-d.ai/role=decode,pod-template-hash=5c77c88766 optimized-baseline-cpu-vllm-decode-5c77c88766-drbrd 1/1 Running 0 5d16h llm-d.ai/accelerator-variant=cpu,llm-d.ai/accelerator-vendor=intel, llm-d.ai/guide=optimized-baseline,llm-d.ai/model=Qwen2.5-0.5B-Instruct, llm-d.ai/role=decode,pod-template-hash=5c77c88766 optimized-baseline-cpu-vllm-decode-5c77c88766-g7kkb 1/1 Running 0 5d16h llm-d.ai/accelerator-variant=cpu,llm-d.ai/accelerator-vendor=intel, llm-d.ai/guide=optimized-baseline,llm-d.ai/model=Qwen2.5-0.5B-Instruct, llm-d.ai/role=decode,pod-template-hash=5c77c88766 quickstart-epp-5f6cbf8c4-tcl7q 2/2 Running 0 17d llm-d-router-gateway=quickstart-epp,llm-d.ai/igw-mode=llm-d-router-gateway, pod-template-hash=5f6cbf8c4 ``` One fleet, no dedicated prefill specialist — every pod both reads prompts and streams answers. That `llm-d.ai/role` label is not cosmetic; it’s the selector slot we use to split responsibilities when we want to. At a high level, running a true P/D split requires two deployments (two fleets): one for prefill and one for decode. Our decode deployment is the default fleet already running. The P/D path simply adds a prefill deployment beside it. Folder structure (guides/recipes excerpt): ```text theme={null} guides/recipes/modelserver/base single-host/ ├─ default/ │ └─ decode-deployment.yaml └─ pd/ └─ base/ └─ prefill-deployment.yaml ``` Each deployment sets the same label key to a different value. In the prefill manifest you will see `llm-d.ai/role: prefill`; the decode manifest keeps `llm-d.ai/role: decode`. The model software is the same — the role label assigns different jobs. ```yaml theme={null} # prefill-deployment.yaml (metadata excerpt) apiVersion: apps/v1 kind: Deployment metadata: name: prefill labels: llm-d.ai/role: prefill spec: selector: matchLabels: llm-d.ai/role: prefill template: metadata: labels: llm-d.ai/role: prefill # ... other fields ``` ```yaml theme={null} # decode-deployment.yaml (metadata excerpt) apiVersion: apps/v1 kind: Deployment metadata: name: decode labels: llm-d.ai/role: decode spec: selector: matchLabels: llm-d.ai/role: decode template: metadata: labels: llm-d.ai/role: decode # ... other fields ``` Second: the decode pods carry a small routing-proxy sidecar added by a patch. That sidecar acts as the receiving dock for saved work coming from a prefill pod. ![A dark-themed diagram titled "A Passenger in Every Decode Pod" showing a Decode Pod with a routing-proxy sidecar ("The receiving dock") connected to a vLLM container. It also shows a patch-sidecar.yaml file noted as "Added by one patch."](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Serving-Bigger-Models-With-llm-d/Demo-Read-the-PD-well-lit-Path-Against-Our-Live-Cluster/decode-pod-routing-proxy-vllm-patch.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=e6e7a97597b62ca5274a4f2cde503408) The sidecar wiring is explicit: it listens on the pod's usual front door (port 8000) while vLLM moves to port 8200. This makes the sidecar capable of proxying normal client traffic and also accepting incoming KV blocks destined for the model. ```yaml theme={null} # decode deployment: routing-proxy sidecar container excerpt apiVersion: apps/v1 kind: Deployment metadata: name: decode spec: template: spec: containers: - name: routing-proxy args: - --port=8000 - --vllm-port=8200 - --kv-connector=nixlv2 - --zap-log-level=1 - --secure-proxy=false image: REPLACE_ROUTING_SIDECAR_IMAGE imagePullPolicy: Always ports: - containerPort: 8000 name: sidecar protocol: TCP # ... resources, securityContext, etc. ``` Third: both deployments belong to a single inference pool (the same chart family as the quickstart release). The pool selector does not match on the `llm-d.ai/role` label; it matches on a shared `llm-d.ai/guide` label. That allows one router to see a single fleet composed of both prefill and decode pods. ```yaml theme={null} router: extraServicePorts: - name: http port: 80 protocol: TCP targetPort: 8081 epp: pluginsConfigFile: "pd-config.yaml" pluginsCustomConfig: pd-config.yaml: | # ... lines 12-43 modelServers: matchLabels: llm-d.ai/guide: "pd-disaggregation" ``` Fourth: the decision brain. The P/D configuration is an EPP (Endpoint Picker) plugin file with two scheduling profiles — `prefill` and `decode`. Filters read the `llm-d.ai/role` label and score candidates differently. A decider plugin can mark appropriate requests as "split", then the prefill profile favors hosts with beneficial cache characteristics while the decode profile favors active-request scores suited for streaming. ```yaml theme={null} epp: pluginsConfigFile: "pd-config.yaml" pluginsCustomConfig: pd-config.yaml: | apiVersion: llm-d.ai/v1alpha1 kind: EndpointPickerConfig plugins: - type: disagg-headers-handler - type: always-disagg-pd-decider - type: disagg-profile-handler parameters: deciderPluginName: always-disagg-pd-decider - type: prefill-filter - type: decode-filter - type: prefix-cache-scorer - type: queue-scorer - type: kv-cache-utilization-scorer - type: active-request-scorer ``` Here’s the exact scheduling-profile scoreboard used in the cluster. The prefill profile weights prefix cache utilization, queue depth, and KV cache utilization more heavily; decode favors active-request scoring plus prefix-cache weight to optimize streaming. ```yaml theme={null} schedulingProfiles: - name: prefill plugins: - pluginRef: prefill-filter - pluginRef: prefix-cache-scorer weight: 3 - pluginRef: queue-scorer weight: 2 - pluginRef: kv-cache-utilization-scorer weight: 2 - name: decode plugins: - pluginRef: decode-filter - pluginRef: active-request-scorer weight: 2 - pluginRef: prefix-cache-scorer weight: 3 ``` Now visualize the runtime flow: a long request hits the gateway. The EPP does double duty — it selects a prefill pod to ingest the full prompt and produce saved work (KV blocks), and a decode pod to receive that saved work and stream the answer token-by-token to the client. The prefill pod consumes the prompt in a single burst and emits KV blocks (saved state). Those blocks are transferred to the chosen decode pod according to the KV transfer connector configured in the prefill deployment's vLLM arguments. ```yaml theme={null} # prefill deployment (modelserver args excerpt) apiVersion: apps/v1 kind: Deployment metadata: name: prefill spec: replicas: 8 template: spec: containers: - name: modelserver command: ["vllm", "serve"] args: - "openai/gpt-oss-120b" - "--disable-access-log-for-endpoints=/health,/metrics,/v1/models" - "--tensor-parallel-size=1" - "--block-size=128" - "--kv-transfer-config" - '{"kv_connector":"NixlConnector", "kv_role":"kv_both"}' - "--no-disable-hybrid-kv-cache-manager" # ... env, resources, probes, volumes ``` What is NIXL? It's NVIDIA's inference transfer (Xfer) library — an open-source, vendor-agnostic data-movement library designed to accelerate point-to-point transfers for distributed AI inference workloads, especially moving large KV caches and supporting disaggregated LLM architectures. For more about Kubernetes networking and orchestration concepts used here, see [Kubernetes Basics](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/). ![An infographic titled "What Is NIXL?" describing NVIDIA Inference Xfer (Transfer) Library with icons and feature boxes labeled "Open source," "Vendor-agnostic," and "Point-to-point." Below it are schematic blocks showing prefill and decode pod memory connected by a "Large KV caches" arrow.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Serving-Bigger-Models-With-llm-d/Demo-Read-the-PD-well-lit-Path-Against-Our-Live-Cluster/nvidia-nixl-inference-transfer-infographic.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=c25f1934c1f74c648e3fa8500c3cc8bb) The prefill keeps `--block-size=128` so KV payloads are chunked predictably. The decode-side sidecar receives KV blocks; the decode pod resumes from the saved state and streams tokens back via the original connection. ![A dark-themed architecture diagram titled "The Stream Comes Back" showing an inference gateway routing user requests to Prefill Pods and Decode Pods (vLLM). It depicts saved work between pods and a sidecar streaming the answer back token-by-token to the user.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Serving-Bigger-Models-With-llm-d/Demo-Read-the-PD-well-lit-Path-Against-Our-Live-Cluster/inference-gateway-prefill-decode-sidecar.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=791c73acbda74e369c3a858c0716937f) Summary — the pieces you can inspect on your cluster: * Role labels (`llm-d.ai/role`) mark prefill vs decode. * A routing-proxy sidecar in decode pods accepts KV transfers and proxies traffic. * The router/EPP uses a plugin config (`pd-config.yaml`) with `prefill` and `decode` scheduling profiles. * Prefill pods use a KV transfer connector (NixlConnector) and shard KV payloads using `--block-size`. Table: key manifest pieces and purpose | Component | Purpose | Example / Manifest location | | - | - | - | | Role label | Marks a pod as `prefill` or `decode` | `llm-d.ai/role: prefill` or `llm-d.ai/role: decode` | | Sidecar | Receives KV blocks and proxies requests | `decode` deployment sidecar container (port 8000) | | Router/EPP | Picks endpoints using P/D scheduling profiles | `pd-config.yaml` (loaded by EPP) | | KV transfer | Point-to-point KV movement between pods | `--kv-transfer-config` args in prefill deployment (`'{"kv_connector":"NixlConnector", "kv_role":"kv_both"}'`) | A production-grade P/D split requires high-bandwidth, low-latency pod-to-pod links (or RDMA-like mechanisms). Software-only P/D shows the control-plane and routing patterns but will not match hardware-backed disaggregation without appropriate networking/hardware. The result: prefill work happens in one place and streaming happens in another, but the manifests, labels, sidecars, and routing glue are all present in the repository and on the cluster. This control-plane + software implementation maps the pieces of a full-scale split without claiming specialized hardware-level performance. If you want to explore further, inspect the `guides/recipes/*/pd` manifests and the `pd-config.yaml` in your quickstart chart to see the exact wiring used by the cluster. > **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/4e52cd5b-6e1d-4d63-af4d-5442e6fa9b81/lesson/d47a2987-79cb-4457-a252-744909f908b6)** This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.