> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # The Cluster Behind Every Demo > Overview of a CPU Kubernetes cluster running llm-d demos, explaining architecture, routing, model serving with vLLM, standalone router, observability with Prometheus and Grafana, and stepwise live traces Welcome to the hands-on half of this course. From here on, every demo runs against a real Kubernetes cluster: real commands, real responses, and the same cluster used across all examples. Before we run anything, getting a clear picture of what’s deployed makes the later demos easier to follow. The cluster consists of four machines: one control-plane node and three worker nodes. ```bash theme={null} $ kubectl get nodes NAME STATUS ROLES AGE VERSION node-cp Ready control-plane 12d v1.31 node-w1 Ready 12d v1.31 node-w2 Ready 12d v1.31 node-w3 Ready 12d v1.31 ``` | Node name | Role | Notes | | - | - | - | | `node-cp` | control-plane | Kubernetes control plane (API server, scheduler, etcd) | | `node-w1` | worker | Runs model servers and monitoring components | | `node-w2` | worker | Runs model servers and monitoring components | | `node-w3` | worker | Runs model servers and monitoring components | There are no GPUs attached — each node is an ordinary CPU-only VM. This choice is intentional: it makes the environment reproducible for anyone with a cloud account or a home lab. The examples focus on how llm-d is architected and operated, not on raw inference throughput. On top of the cluster runs llm-d, installed using the project's Quick Start along the optimized baseline path. The demo environment uses the "one pool" configuration: a single pool of identical servers with cache-aware routing enabled. The specific model served in this environment is a half‑billion‑parameter instruct variant from the Qwen family, labeled `qwen2.5-0.5b-instruct`. ![A dark-themed infographic titled "What's Running on It" showing a quickstart menu with options like "Optimized baseline" and a boxed "One pool" containing three vLLM instances. Below it is an ilm-d layer and four labeled nodes (node-cp, node-w1, node-w2, node-w3) representing a cluster.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Tour-the-Running-Stack/The-Cluster-Behind-Every-Demo/optimized-baseline-one-pool-vllm-cluster.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=6cfcd3a4253f3b0d5d950944fb15c028) Remember the model-size discussion earlier: a half‑billion-parameter model typically compresses to around a gigabyte on disk, which is why a CPU machine can serve it reasonably well. To give an intuitive feel for what model weight data looks like, here’s a tiny illustrative excerpt (not real weights): ```text theme={null} qwen2.5-0.5b-instruct 0.42 1.7 0.42 -1.7 0.42 1.7 0.42 -1.7 0.42 1.7 0.42 -1.7 ``` This environment runs three replicas of the model server — three pods, each loading the entire model into memory. That small fleet creates real routing decisions for the router layer. The router in this setup runs llm-d's standalone mode. Normally, llm-d separates a gateway (proxy) and an endpoint picker into different deployments. In standalone mode the Envoy proxy (which holds client connections) and the endpoint picker (EPP) run together inside the same pod. In short: there is no separate gateway deployment in front of the EPP. ![A diagram titled "Standalone Mode" shows a user icon connected to "the EPP's own pod." Inside the pod are two components labeled "Proxy (Envoy)" and "Picker – EPP."](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Tour-the-Running-Stack/The-Cluster-Behind-Every-Demo/epp-standalone-pod-proxy-picker.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=a206200f45c793ab04abbde62c3b225d) Alongside the serving layer we run a lightweight monitoring stack (Prometheus + Grafana). Many demos in this lesson rely on inspecting llm-d’s metrics and dashboards to illustrate behavior during routing, cache hits, and rollouts. ![A dark-themed diagram titled "The Monitoring Stack" showing stacked vLLM instance blocks on the left and two monitoring tool icons (Prometheus and Grafana) on the right. Dotted boxes group the components and a small "© Copyright KodeKloud" appears in the bottom-left.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Tour-the-Running-Stack/The-Cluster-Behind-Every-Demo/vllm-monitoring-stack-prometheus-grafana.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=a58d14dff42657e3810c8b3b124538cb) Nothing in this lesson was installed live during recording — the cluster was pre-built. Model servers require time to start because they must pull all weights into memory before serving requests, so watching weight pulls adds little to the demonstration. If you want to reproduce this environment, run the project's Quick Start exactly as used to create the demo cluster: ```bash theme={null} llm-d.ai - quickstart ``` If you plan to recreate the cluster, follow the Quick Start instructions step-by-step. The Quick Start sets up the control plane, worker nodes, llm-d, and the optimized baseline configuration so your environment matches these demos. One rule governs every demo here: this cluster demonstrates how llm-d is designed and behaves, not how fast it can be. Control plane behavior, routing logic, Kubernetes objects, metrics, and rollout procedures are all faithfully represented and reproducible on CPU-only hardware. Raw throughput, however, depends on GPU acceleration and is out of scope for this CPU-based environment. This CPU cluster is not a benchmark for production performance. When raw speed matters, GPU-based benchmarks from the project will be shown explicitly and labeled accordingly. The pedagogical advantage of the slower, CPU-based setup is visibility: you can observe pauses, streaming responses, and roundtrips at a human pace — details that blur on fast GPU hardware. Lesson plan (high level) * Full tour: map every llm-d Kubernetes object back to the conceptual architecture you saw earlier. * Live request: send a real request and observe a real answer. * Trace: slow down and follow a single request through each phase (routing, queueing, endpoint selection, model inference, and response streaming). ![A presentation slide titled "The Honest Rule" with a scales icon at the top. Two boxed lists below: a green box showing what it proves (routing, objects, metrics, rollouts) and a blue box showing what it cannot prove (raw speed).](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Tour-the-Running-Stack/The-Cluster-Behind-Every-Demo/honest-rule-proves-routing-objects-rollouts.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=8c29067e2983b585ef349ab0093ed7df) To help you map components to concepts, here’s a quick reference: | Component | Purpose | Conceptual mapping | | - | - | - | | Model pods (vLLM) | Serve model inference | Model execution units | | Pool | Group of identical servers | Capacity and routing domain | | Endpoint picker (EPP) | Choose a serving endpoint | Routing + load distribution | | Envoy proxy | Accept and maintain client connections | Gateway / connection front-end | | Prometheus + Grafana | Collect and visualize metrics | Observability and monitoring | We’ll proceed from high-level mapping down to a live request and complete trace. ![A simple architecture diagram titled "Where We're Headed" showing components (Pods, Pool, Endpoint Picker, Proxy) on the left feeding into a Cluster icon, which then outputs an Answer (green check).](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Tour-the-Running-Stack/The-Cluster-Behind-Every-Demo/cluster-pods-pool-endpoint-proxy-answer.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=8f6e3c68897240f69217fadd7a8cdb43) Links and references * Quick Start and installation: `https://llm-d.ai/docs/quickstart` * Kubernetes basics: [https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/) * Prometheus: [https://prometheus.io/](https://prometheus.io/) * Grafana: [https://grafana.com/](https://grafana.com/) > **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/cb6d2710-3350-448e-bc67-d78835a558a9/lesson/e5e93642-d3fe-4e62-b52a-09932fe5ba9f)** This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.