Skip to main content
Welcome to the hands-on half of this course. From here on, every demo runs against a real Kubernetes cluster: real commands, real responses, and the same cluster used across all examples. Before we run anything, getting a clear picture of what’s deployed makes the later demos easier to follow. The cluster consists of four machines: one control-plane node and three worker nodes.
There are no GPUs attached — each node is an ordinary CPU-only VM. This choice is intentional: it makes the environment reproducible for anyone with a cloud account or a home lab. The examples focus on how llm-d is architected and operated, not on raw inference throughput. On top of the cluster runs llm-d, installed using the project’s Quick Start along the optimized baseline path. The demo environment uses the “one pool” configuration: a single pool of identical servers with cache-aware routing enabled. The specific model served in this environment is a half‑billion‑parameter instruct variant from the Qwen family, labeled qwen2.5-0.5b-instruct.
A dark-themed infographic titled "What's Running on It" showing a quickstart menu with options like "Optimized baseline" and a boxed "One pool" containing three vLLM instances. Below it is an ilm-d layer and four labeled nodes (node-cp, node-w1, node-w2, node-w3) representing a cluster.
Remember the model-size discussion earlier: a half‑billion-parameter model typically compresses to around a gigabyte on disk, which is why a CPU machine can serve it reasonably well. To give an intuitive feel for what model weight data looks like, here’s a tiny illustrative excerpt (not real weights):
This environment runs three replicas of the model server — three pods, each loading the entire model into memory. That small fleet creates real routing decisions for the router layer. The router in this setup runs llm-d’s standalone mode. Normally, llm-d separates a gateway (proxy) and an endpoint picker into different deployments. In standalone mode the Envoy proxy (which holds client connections) and the endpoint picker (EPP) run together inside the same pod. In short: there is no separate gateway deployment in front of the EPP.
A diagram titled "Standalone Mode" shows a user icon connected to "the EPP's own pod." Inside the pod are two components labeled "Proxy (Envoy)" and "Picker – EPP."
Alongside the serving layer we run a lightweight monitoring stack (Prometheus + Grafana). Many demos in this lesson rely on inspecting llm-d’s metrics and dashboards to illustrate behavior during routing, cache hits, and rollouts.
A dark-themed diagram titled "The Monitoring Stack" showing stacked vLLM instance blocks on the left and two monitoring tool icons (Prometheus and Grafana) on the right. Dotted boxes group the components and a small "© Copyright KodeKloud" appears in the bottom-left.
Nothing in this lesson was installed live during recording — the cluster was pre-built. Model servers require time to start because they must pull all weights into memory before serving requests, so watching weight pulls adds little to the demonstration. If you want to reproduce this environment, run the project’s Quick Start exactly as used to create the demo cluster:
If you plan to recreate the cluster, follow the Quick Start instructions step-by-step. The Quick Start sets up the control plane, worker nodes, llm-d, and the optimized baseline configuration so your environment matches these demos.
One rule governs every demo here: this cluster demonstrates how llm-d is designed and behaves, not how fast it can be. Control plane behavior, routing logic, Kubernetes objects, metrics, and rollout procedures are all faithfully represented and reproducible on CPU-only hardware. Raw throughput, however, depends on GPU acceleration and is out of scope for this CPU-based environment.
This CPU cluster is not a benchmark for production performance. When raw speed matters, GPU-based benchmarks from the project will be shown explicitly and labeled accordingly.
The pedagogical advantage of the slower, CPU-based setup is visibility: you can observe pauses, streaming responses, and roundtrips at a human pace — details that blur on fast GPU hardware. Lesson plan (high level)
  • Full tour: map every llm-d Kubernetes object back to the conceptual architecture you saw earlier.
  • Live request: send a real request and observe a real answer.
  • Trace: slow down and follow a single request through each phase (routing, queueing, endpoint selection, model inference, and response streaming).
A presentation slide titled "The Honest Rule" with a scales icon at the top. Two boxed lists below: a green box showing what it proves (routing, objects, metrics, rollouts) and a blue box showing what it cannot prove (raw speed).
To help you map components to concepts, here’s a quick reference: We’ll proceed from high-level mapping down to a live request and complete trace.
A simple architecture diagram titled "Where We're Headed" showing components (Pods, Pool, Endpoint Picker, Proxy) on the left feeding into a Cluster icon, which then outputs an Answer (green check).
Links and references

Watch Video