> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# What This Course Covers

> Course teaching practical engineering for serving large language models at scale with llm-d on Kubernetes, explaining challenges, architecture, demos, and deployment best practices

This course answers one practical engineering question:

What does it really take to serve a large language model — the kind of system that powers services like [ChatGPT](https://chat.openai.com/) and [Claude](https://www.anthropic.com/) — to many users at once?

If you’ve run apps on Kubernetes before, the short answer may sound familiar: package the app, put a load balancer in front, and add replicas as traffic grows. But serving large language models (LLMs) breaks that familiar playbook in surprising ways. This course explains exactly where the classic approach fails, why it fails, and what to do about it.

The practical solution we’ll converge on is called llm-d: an open-source project for serving language models across many machines. It runs on Kubernetes — the platform you probably already know — and it’s the destination of this course. But we won’t begin with the tool. You need to feel the problem first, so the solution becomes the natural next step.

The course flows in two halves.

The first half is the story. There’s no cluster or terminal — just concepts, built step-by-step. We start with something you use every day: asking a chatbot a question. From there we descend to the hardware underneath and then climb back up until we assemble the full system. Each new idea is presented as the answer to a practical problem created by the previous idea, so nothing should ever feel like

![A dark-themed slide titled "Part 1 – The Story" with a flowchart showing a user feeding into a Chatbot that links to a Model, Servers, and Hardware. To the right is a separate vertical sequence of boxes labeled "Problem → Idea → Problem."](https://mintcdn.com/kodekloud-c4ac6d9a/HJ60thRsNzBTyswi/images/llm-d-A-Deep-Dive/From-ChatGPT-to-a-Model-Server/What-This-Course-Covers/part1-story-user-chatbot-model-hardware.jpg?fit=max\&auto=format\&n=HJ60thRsNzBTyswi\&q=85\&s=d89e99043b29a117692a8c0602fa9313)

it fell out of the sky.

By the end of that half, llm-d will feel like the obvious conclusion rather than a tool to memorize.

The second half is demo-driven. Everything there runs against a real Kubernetes cluster, with real commands and live demos that show what actually happens in production-like conditions:

```bash theme={null}
$ kubectl
```

All demos are executed live so you can see the system behavior and reproduce the steps on your own cluster when you’re ready.

Meet the cluster we use for demos. It was prepared before the lessons so no installation is required during sessions — you only need the course material to follow along.

* Cluster layout: 4 nodes total — 1 control plane + 3 worker nodes.
* Models: One small model, Qwen2.5-0.5B-Instruct, deployed as three replicas.
* Routing: One llm-d router that decides where each request goes.

![A titled diagram "Meet the Cluster" showing a computing cluster with a control plane and three nodes (Node 01, Node 02, Node 03) represented by hexagon icons. A labeled component "Qwen2.5-0.5B-Instruct" is shown inside the dashed cluster boundary.](https://mintcdn.com/kodekloud-c4ac6d9a/HJ60thRsNzBTyswi/images/llm-d-A-Deep-Dive/From-ChatGPT-to-a-Model-Server/What-This-Course-Covers/qwen2-5-0-5b-instruct-cluster.jpg?fit=max\&auto=format\&n=HJ60thRsNzBTyswi\&q=85\&s=d459abd1a678fc2336150660265adf9f)

Everything you’ll see in demos was produced on this example cluster. If you want to run your own copy later, the [llm-d Quick Start Guide](https://github.com/llm-d/llm-d) contains the exact recipe used to build this environment and can run on hardware you may already have.

This course uses a small, reproducible cluster so the lessons stay affordable and easy to follow. When we need GPU-level performance, we reference llm-d’s published GPU benchmarks instead of implying the CPU cluster is equally fast.

One honest note up front: the demo cluster runs on ordinary CPUs, not GPUs — the specialized processors typically used for large models. We’ll explain the GPU architecture, the role of VRAM, and why GPUs matter for scale in a dedicated section.

![A dark-themed infographic titled "An Honest Note" comparing Ordinary CPUs on the left (icons of cores and a checkmark) with GPUs on the right (a graphics-card illustration labeled "VRAM" and a red X).](https://mintcdn.com/kodekloud-c4ac6d9a/HJ60thRsNzBTyswi/images/llm-d-A-Deep-Dive/From-ChatGPT-to-a-Model-Server/What-This-Course-Covers/honest-note-cpu-vs-gpu-vram.jpg?fit=max\&auto=format\&n=HJ60thRsNzBTyswi\&q=85\&s=b2b7dce598bebf7f575a84744ce1bef8)

For transparency, when we run the llm-d quickstart on this cluster you’ll see output like:

```bash theme={null}
$ llm-d quickstart
✓ Cluster is ready
```

We’ll always indicate which hardware produced the numbers and dashboards you see.

What you’ll be able to do by the end of the course

| Outcome | What you’ll learn |
| - | - |
| Model-serving fundamentals | Explain why ordinary load balancing fails for LLMs and how serving requirements differ from typical web services |
| llm-d components | Name each llm-d component and describe its responsibility (router, model servers, metrics, etc.) |
| Observability & routing | Use live metrics to improve routing decisions and react to resource pressure |
| Safe updates | Roll out model or infrastructure updates under traffic without dropping requests |
| Validation | Read dashboards and benchmarks to assess system health and performance |

Up next: we start at the very beginning. We’ll take the chatbots you use every day—ChatGPT or Claude—pull them apart conceptually, and rebuild a production-capable model-serving system from the hardware upward.

Links and references

* [llm-d Quick Start Guide](https://github.com/llm-d/llm-d)
* [Kubernetes Basics](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/)
* [Understanding GPUs for ML](https://developer.nvidia.com/deep-learning)

> **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/b16615f2-d3fb-42ec-8823-f12ef7e820c6/lesson/7e30eacc-a9ce-4e0f-9eb4-1a2667ac1a2b)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.