Skip to main content
This course answers one practical engineering question: What does it really take to serve a large language model — the kind of system that powers services like ChatGPT and Claude — to many users at once? If you’ve run apps on Kubernetes before, the short answer may sound familiar: package the app, put a load balancer in front, and add replicas as traffic grows. But serving large language models (LLMs) breaks that familiar playbook in surprising ways. This course explains exactly where the classic approach fails, why it fails, and what to do about it. The practical solution we’ll converge on is called llm-d: an open-source project for serving language models across many machines. It runs on Kubernetes — the platform you probably already know — and it’s the destination of this course. But we won’t begin with the tool. You need to feel the problem first, so the solution becomes the natural next step. The course flows in two halves. The first half is the story. There’s no cluster or terminal — just concepts, built step-by-step. We start with something you use every day: asking a chatbot a question. From there we descend to the hardware underneath and then climb back up until we assemble the full system. Each new idea is presented as the answer to a practical problem created by the previous idea, so nothing should ever feel like
A dark-themed slide titled "Part 1 – The Story" with a flowchart showing a user feeding into a Chatbot that links to a Model, Servers, and Hardware. To the right is a separate vertical sequence of boxes labeled "Problem → Idea → Problem."
it fell out of the sky. By the end of that half, llm-d will feel like the obvious conclusion rather than a tool to memorize. The second half is demo-driven. Everything there runs against a real Kubernetes cluster, with real commands and live demos that show what actually happens in production-like conditions:
All demos are executed live so you can see the system behavior and reproduce the steps on your own cluster when you’re ready. Meet the cluster we use for demos. It was prepared before the lessons so no installation is required during sessions — you only need the course material to follow along.
  • Cluster layout: 4 nodes total — 1 control plane + 3 worker nodes.
  • Models: One small model, Qwen2.5-0.5B-Instruct, deployed as three replicas.
  • Routing: One llm-d router that decides where each request goes.
A titled diagram "Meet the Cluster" showing a computing cluster with a control plane and three nodes (Node 01, Node 02, Node 03) represented by hexagon icons. A labeled component "Qwen2.5-0.5B-Instruct" is shown inside the dashed cluster boundary.
Everything you’ll see in demos was produced on this example cluster. If you want to run your own copy later, the llm-d Quick Start Guide contains the exact recipe used to build this environment and can run on hardware you may already have.
This course uses a small, reproducible cluster so the lessons stay affordable and easy to follow. When we need GPU-level performance, we reference llm-d’s published GPU benchmarks instead of implying the CPU cluster is equally fast.
One honest note up front: the demo cluster runs on ordinary CPUs, not GPUs — the specialized processors typically used for large models. We’ll explain the GPU architecture, the role of VRAM, and why GPUs matter for scale in a dedicated section.
A dark-themed infographic titled "An Honest Note" comparing Ordinary CPUs on the left (icons of cores and a checkmark) with GPUs on the right (a graphics-card illustration labeled "VRAM" and a red X).
For transparency, when we run the llm-d quickstart on this cluster you’ll see output like:
We’ll always indicate which hardware produced the numbers and dashboards you see. What you’ll be able to do by the end of the course Up next: we start at the very beginning. We’ll take the chatbots you use every day—ChatGPT or Claude—pull them apart conceptually, and rebuild a production-capable model-serving system from the hardware upward. Links and references

Watch Video