Skip to main content

llm-d: A Deep Dive

Guide to building scalable, observable Kubernetes architectures and operational practices for serving large language models, covering failure modes, routing, placement, rollout strategies, and hands-on demos.

Notes for the llm-d: A Deep Dive course: 43 lessons across 11 sections.

From ChatGPT to a Model Server

Inside a Single Request

From One Server to a Fleet

Meet llm-d

Tour the Running Stack

Smart Routing, Live

Getting More From the KV Cache

Serving Bigger Models With llm-d

Running It Day to Day

Seeing What Is Happening

Wrap Up: Agents and Beyond