Demo Read the PD well lit Path Against Our Live Cluster

Earlier we discussed why the largest fleets separate prefill from decode: the two phases stress hardware in opposite ways. A big read (prefill) moves large amounts of prompt context through memory at once, while streaming (decode) emits tokens one-by-one. Splitting those roles into dedicated pools lets you tune prefill and decode independently — in practice this can yield significantly higher tokens-per-second on the same hardware. Long prompts, mixed traffic, and lock-step expert groups are workloads where the split delivers the most benefit.

Splitting prefill and decode typically improves throughput because prefill optimizes bulk memory access and cache warm-up, while decode optimizes per-token latency and streaming behavior.

Why not run a full hardware-backed split in this demo? A true, fully disaggregated P/D system depends on extremely fast pod-to-pod links and memory-visible transfers across hosts — hardware our small CPU cluster doesn’t provide. Instead of faking a hardware topology, we inspect what the cluster actually runs, read the real P/D manifests, and show how close a control-plane + software P/D path can get to a full split. The code, labels, sidecars, and scheduler wiring are all present; we simply avoid claiming hardware-level transfer performance we don’t have. Start by listing pod labels in the llm-d namespace:

$ kubectl get pods -n llm-d --show-labels

Look at the label on the model pods: llm-d.ai/role=decode. Every model pod currently uses that value — a single fleet handling both prompt-reading and streaming.

$ kubectl get pods -n llm-d --show-labels
NAME                                                   READY   STATUS    RESTARTS   AGE
optimized-baseline-cpu-vllm-decode-5c77c88766-4hmpb    1/1     Running   0          5d16h
  llm-d.ai/accelerator-variant=cpu,llm-d.ai/accelerator-vendor=intel,
  llm-d.ai/guide=optimized-baseline,llm-d.ai/model=Qwen2.5-0.5B-Instruct,
  llm-d.ai/role=decode,pod-template-hash=5c77c88766
optimized-baseline-cpu-vllm-decode-5c77c88766-drbrd    1/1     Running   0          5d16h
  llm-d.ai/accelerator-variant=cpu,llm-d.ai/accelerator-vendor=intel,
  llm-d.ai/guide=optimized-baseline,llm-d.ai/model=Qwen2.5-0.5B-Instruct,
  llm-d.ai/role=decode,pod-template-hash=5c77c88766
optimized-baseline-cpu-vllm-decode-5c77c88766-g7kkb    1/1     Running   0          5d16h
  llm-d.ai/accelerator-variant=cpu,llm-d.ai/accelerator-vendor=intel,
  llm-d.ai/guide=optimized-baseline,llm-d.ai/model=Qwen2.5-0.5B-Instruct,
  llm-d.ai/role=decode,pod-template-hash=5c77c88766
quickstart-epp-5f6cbf8c4-tcl7q                         2/2     Running   0          17d
  llm-d-router-gateway=quickstart-epp,llm-d.ai/igw-mode=llm-d-router-gateway,
  pod-template-hash=5f6cbf8c4

One fleet, no dedicated prefill specialist — every pod both reads prompts and streams answers. That llm-d.ai/role label is not cosmetic; it’s the selector slot we use to split responsibilities when we want to. At a high level, running a true P/D split requires two deployments (two fleets): one for prefill and one for decode. Our decode deployment is the default fleet already running. The P/D path simply adds a prefill deployment beside it. Folder structure (guides/recipes excerpt):

guides/recipes/modelserver/base
single-host/
├─ default/
│  └─ decode-deployment.yaml
└─ pd/
   └─ base/
      └─ prefill-deployment.yaml

Each deployment sets the same label key to a different value. In the prefill manifest you will see llm-d.ai/role: prefill; the decode manifest keeps llm-d.ai/role: decode. The model software is the same — the role label assigns different jobs.

# prefill-deployment.yaml (metadata excerpt)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefill
  labels:
    llm-d.ai/role: prefill
spec:
  selector:
    matchLabels:
      llm-d.ai/role: prefill
  template:
    metadata:
      labels:
        llm-d.ai/role: prefill
# ... other fields

# decode-deployment.yaml (metadata excerpt)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: decode
  labels:
    llm-d.ai/role: decode
spec:
  selector:
    matchLabels:
      llm-d.ai/role: decode
  template:
    metadata:
      labels:
        llm-d.ai/role: decode
# ... other fields

Second: the decode pods carry a small routing-proxy sidecar added by a patch. That sidecar acts as the receiving dock for saved work coming from a prefill pod.

A dark-themed diagram titled "A Passenger in Every Decode Pod" showing a Decode Pod with a routing-proxy sidecar ("The receiving dock") connected to a vLLM container. It also shows a patch-sidecar.yaml file noted as "Added by one patch."

The sidecar wiring is explicit: it listens on the pod’s usual front door (port 8000) while vLLM moves to port 8200. This makes the sidecar capable of proxying normal client traffic and also accepting incoming KV blocks destined for the model.

# decode deployment: routing-proxy sidecar container excerpt
apiVersion: apps/v1
kind: Deployment
metadata:
  name: decode
spec:
  template:
    spec:
      containers:
      - name: routing-proxy
        args:
        - --port=8000
        - --vllm-port=8200
        - --kv-connector=nixlv2
        - --zap-log-level=1
        - --secure-proxy=false
        image: REPLACE_ROUTING_SIDECAR_IMAGE
        imagePullPolicy: Always
        ports:
        - containerPort: 8000
          name: sidecar
          protocol: TCP
      # ... resources, securityContext, etc.

Third: both deployments belong to a single inference pool (the same chart family as the quickstart release). The pool selector does not match on the llm-d.ai/role label; it matches on a shared llm-d.ai/guide label. That allows one router to see a single fleet composed of both prefill and decode pods.

router:
  extraServicePorts:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8081

epp:
  pluginsConfigFile: "pd-config.yaml"
  pluginsCustomConfig:
    pd-config.yaml: |
      # ... lines 12-43
modelServers:
  matchLabels:
    llm-d.ai/guide: "pd-disaggregation"

Fourth: the decision brain. The P/D configuration is an EPP (Endpoint Picker) plugin file with two scheduling profiles — prefill and decode. Filters read the llm-d.ai/role label and score candidates differently. A decider plugin can mark appropriate requests as “split”, then the prefill profile favors hosts with beneficial cache characteristics while the decode profile favors active-request scores suited for streaming.

epp:
  pluginsConfigFile: "pd-config.yaml"
  pluginsCustomConfig:
    pd-config.yaml: |
      apiVersion: llm-d.ai/v1alpha1
      kind: EndpointPickerConfig
      plugins:
      - type: disagg-headers-handler
      - type: always-disagg-pd-decider
      - type: disagg-profile-handler
        parameters:
          deciderPluginName: always-disagg-pd-decider
      - type: prefill-filter
      - type: decode-filter
      - type: prefix-cache-scorer
      - type: queue-scorer
      - type: kv-cache-utilization-scorer
      - type: active-request-scorer

Here’s the exact scheduling-profile scoreboard used in the cluster. The prefill profile weights prefix cache utilization, queue depth, and KV cache utilization more heavily; decode favors active-request scoring plus prefix-cache weight to optimize streaming.

schedulingProfiles:
- name: prefill
  plugins:
  - pluginRef: prefill-filter
  - pluginRef: prefix-cache-scorer
    weight: 3
  - pluginRef: queue-scorer
    weight: 2
  - pluginRef: kv-cache-utilization-scorer
    weight: 2
- name: decode
  plugins:
  - pluginRef: decode-filter
  - pluginRef: active-request-scorer
    weight: 2
  - pluginRef: prefix-cache-scorer
    weight: 3

Now visualize the runtime flow: a long request hits the gateway. The EPP does double duty — it selects a prefill pod to ingest the full prompt and produce saved work (KV blocks), and a decode pod to receive that saved work and stream the answer token-by-token to the client. The prefill pod consumes the prompt in a single burst and emits KV blocks (saved state). Those blocks are transferred to the chosen decode pod according to the KV transfer connector configured in the prefill deployment’s vLLM arguments.

# prefill deployment (modelserver args excerpt)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefill
spec:
  replicas: 8
  template:
    spec:
      containers:
      - name: modelserver
        command: ["vllm", "serve"]
        args:
        - "openai/gpt-oss-120b"
        - "--disable-access-log-for-endpoints=/health,/metrics,/v1/models"
        - "--tensor-parallel-size=1"
        - "--block-size=128"
        - "--kv-transfer-config"
        - '{"kv_connector":"NixlConnector", "kv_role":"kv_both"}'
        - "--no-disable-hybrid-kv-cache-manager"
      # ... env, resources, probes, volumes

What is NIXL? It’s NVIDIA’s inference transfer (Xfer) library — an open-source, vendor-agnostic data-movement library designed to accelerate point-to-point transfers for distributed AI inference workloads, especially moving large KV caches and supporting disaggregated LLM architectures. For more about Kubernetes networking and orchestration concepts used here, see Kubernetes Basics.

An infographic titled "What Is NIXL?" describing NVIDIA Inference Xfer (Transfer) Library with icons and feature boxes labeled "Open source," "Vendor-agnostic," and "Point-to-point." Below it are schematic blocks showing prefill and decode pod memory connected by a "Large KV caches" arrow.

The prefill keeps --block-size=128 so KV payloads are chunked predictably. The decode-side sidecar receives KV blocks; the decode pod resumes from the saved state and streams tokens back via the original connection.

A dark-themed architecture diagram titled "The Stream Comes Back" showing an inference gateway routing user requests to Prefill Pods and Decode Pods (vLLM). It depicts saved work between pods and a sidecar streaming the answer back token-by-token to the user.

Summary — the pieces you can inspect on your cluster:

Role labels (llm-d.ai/role) mark prefill vs decode.
A routing-proxy sidecar in decode pods accepts KV transfers and proxies traffic.
The router/EPP uses a plugin config (pd-config.yaml) with prefill and decode scheduling profiles.
Prefill pods use a KV transfer connector (NixlConnector) and shard KV payloads using --block-size.

Table: key manifest pieces and purpose

Component	Purpose	Example / Manifest location
Role label	Marks a pod as `prefill` or `decode`	`llm-d.ai/role: prefill` or `llm-d.ai/role: decode`
Sidecar	Receives KV blocks and proxies requests	`decode` deployment sidecar container (port 8000)
Router/EPP	Picks endpoints using P/D scheduling profiles	`pd-config.yaml` (loaded by EPP)
KV transfer	Point-to-point KV movement between pods	`--kv-transfer-config` args in prefill deployment (`'{"kv_connector":"NixlConnector", "kv_role":"kv_both"}'`)

A production-grade P/D split requires high-bandwidth, low-latency pod-to-pod links (or RDMA-like mechanisms). Software-only P/D shows the control-plane and routing patterns but will not match hardware-backed disaggregation without appropriate networking/hardware.

The result: prefill work happens in one place and streaming happens in another, but the manifests, labels, sidecars, and routing glue are all present in the repository and on the cluster. This control-plane + software implementation maps the pieces of a full-scale split without claiming specialized hardware-level performance. If you want to explore further, inspect the guides/recipes/*/pd manifests and the pd-config.yaml in your quickstart chart to see the exact wiring used by the cluster.

From ChatGPT to a Model Server

Inside a Single Request

From One Server to a Fleet

Meet llm-d

Tour the Running Stack

Smart Routing, Live

Getting More From the KV Cache

Serving Bigger Models With llm-d

Running It Day to Day

Seeing What Is Happening

Wrap Up: Agents and Beyond

Demo Read the PD well lit Path Against Our Live Cluster

Watch Video