From ChatGPT to a Model Server
- Introduction
- What This Course Covers
- What a Model Really Is
- The Hardware CPU GPU and VRAM
- From a Script to a Service
- A Closer Look at the Model Server
Inside a Single Request
From One Server to a Fleet
Meet llm-d
- Meet llm d
- Putting It to Work
- Inside the Cluster
- How the Router Decides
- How the Router Knows
- Routing on a Prediction
- When the Scratchpad Overflows
Tour the Running Stack
- The Cluster Behind Every Demo
- Demo Tour the Running Stack and Send the First Request
- Demo Watch One Request Happen
Smart Routing, Live
- The Scorers Behind Every Decision
- Demo The Same Prompt Lands on the Same Server
- Demo Cold Requests Spread Out
Getting More From the KV Cache
- Precise Prefix Cache Routing
- Tiered Cache Spill Over to CPU and Disk
- Demo A Multi Turn Chat Reuses the Cache
Serving Bigger Models With llm-d
- Wide Expert Parallelism
- Demo Read the PD well lit Path Against Our Live Cluster
- What Really Needs GPUs
Running It Day to Day
- Flow Control A Line for Requests
- Autoscaling on the Right Signals
- Rollouts With No Downtime
- Demo A Rolling Update Under Live Traffic