# KodeKloud: Courses llm-d: A Deep Dive ## llm-d: A Deep Dive ### From ChatGPT to a Model Server - [Course: llm-d: A Deep Dive](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/index.md): Guide to building scalable, observable Kubernetes architectures and operational practices for serving large language models, covering failure modes, routing, placement, rollout strategies, and hands-on demos. - [Introduction](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/From-ChatGPT-to-a-Model-Server/Introduction/page.md): Guide to building scalable, observable Kubernetes architectures and operational practices for serving large language models, covering failure modes, routing, placement, rollout strategies, and hands-on demos. - [What This Course Covers](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/From-ChatGPT-to-a-Model-Server/What-This-Course-Covers/page.md): Course teaching practical engineering for serving large language models at scale with llm-d on Kubernetes, explaining challenges, architecture, demos, and deployment best practices - [What a Model Really Is](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/From-ChatGPT-to-a-Model-Server/What-a-Model-Really-Is/page.md): Explains what a language model is, how Transformers and learned weights power model behavior, and why large models require significant memory and compute and are served via model servers - [The Hardware CPU GPU and VRAM](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/From-ChatGPT-to-a-Model-Server/The-Hardware-CPU-GPU-and-VRAM/page.md): Explains why GPUs with VRAM are needed for large models, focusing on compute, memory bandwidth, and capacity tradeoffs. - [From a Script to a Service](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/From-ChatGPT-to-a-Model-Server/From-a-Script-to-a-Service/page.md): Explains turning a PyTorch model script into an always-on GPU model server with vLLM providing batching, scheduling, memory management and OpenAI compatible API - [A Closer Look at the Model Server](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/From-ChatGPT-to-a-Model-Server/A-Closer-Look-at-the-Model-Server/page.md): Explains what a model server does, how vLLM loads models into GPU memory, impacts latency, startup time, scaling costs, and implications for concurrency and autoscaling. ### Inside a Single Request - [How a Model Writes Text](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Inside-a-Single-Request/How-a-Model-Writes-Text/page.md): Explains how autoregressive language models tokenize text, perform initial context encoding and per-token generation, and how token counts affect latency, cost, and serving complexity - [The Pause and the Stream](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Inside-a-Single-Request/The-Pause-and-the-Stream/page.md): Explains why LLM responses show an initial pause then steady streaming by describing prefill and decode phases and their differing compute versus memory bandwidth bottlenecks. - [The Speed Limit](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Inside-a-Single-Request/The-Speed-Limit/page.md): How GPU memory bandwidth limits transformer decode throughput and how prefill versus decode, KV caching, and incremental decoding optimize latency and token generation - [The Models Scratchpad](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Inside-a-Single-Request/The-Models-Scratchpad/page.md): Explains how models use prefill, KV cache, and prefix caching to reuse computation, reduce per-token cost, and trade GPU memory for lower latency and higher throughput. ### From One Server to a Fleet - [One GPU Many Users](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/From-One-Server-to-a-Fleet/One-GPU-Many-Users/page.md): How to serve many users from one GPU by batching requests, managing VRAM limited KV caches, and using sharding, offload, and quantization to increase concurrency - [Serving the Giants](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/From-One-Server-to-a-Fleet/Serving-the-Giants/page.md): Describes sharding giant transformer models across multiple GPUs using pipeline and tensor parallelism, illustrating tradeoffs, communication patterns, and performance considerations for distributed inference. - [One Machine or Many](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/From-One-Server-to-a-Fleet/One-Machine-or-Many/page.md): Describes model sharding choices, tradeoffs between tensor and pipeline parallelism based on interconnect speed, and routing plus cache strategies for multi server deployments - [Why Load Balancing Breaks](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/From-One-Server-to-a-Fleet/Why-Load-Balancing-Breaks/page.md): Explains why traditional load balancers fail for LLM fleets and advocates LLM-aware routing considering KV cache locality, request cost, and server telemetry to reduce latency ### Meet llm-d - [Meet llm d](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Meet-llm-d/Meet-llm-d/page.md): An open source smart router for LLM serving that routes by KV cache and server capacity, splitting prefill and decode pools to improve throughput, latency, and resource utilization - [Putting It to Work](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Meet-llm-d/Putting-It-to-Work/page.md): Guide to deploying llm-d, tuning routing, caching, pool splits, batching and model sharding with three recommended production deployment paths and starting defaults - [Inside the Cluster](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Meet-llm-d/Inside-the-Cluster/page.md): Explains llm-d's Kubernetes architecture, components, routing, sharded vLLM deployments, and how a gateway and scheduler route requests to prefill and decode pods for distributed inference. - [How the Router Decides](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Meet-llm-d/How-the-Router-Decides/page.md): Explains how the Endpoint Picker scores and selects pods using cache affinity and load-awareness to reduce context reloads and improve LLM serving throughput. - [How the Router Knows](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Meet-llm-d/How-the-Router-Knows/page.md): Explains how a router uses Kubernetes API watches and per-pod metrics scrapes plus Envoy ext_proc to score and select pods for low-latency request routing - [Routing on a Prediction](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Meet-llm-d/Routing-on-a-Prediction/page.md): Describes predictive routing that predicts per-pod TTFT and TPOT with XGBoost trained on live requests to minimize latency, noting KV cache eviction limits effectiveness. - [When the Scratchpad Overflows](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Meet-llm-d/When-the-Scratchpad-Overflows/page.md): Tiered caching and live block indexing that move LLM conversation state between GPU RAM and disk to avoid recomputation and enable reliable routing. ### Tour the Running Stack - [The Cluster Behind Every Demo](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Tour-the-Running-Stack/The-Cluster-Behind-Every-Demo/page.md): Overview of a CPU Kubernetes cluster running llm-d demos, explaining architecture, routing, model serving with vLLM, standalone router, observability with Prometheus and Grafana, and stepwise live traces - [Demo Tour the Running Stack and Send the First Request](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Tour-the-Running-Stack/Demo-Tour-the-Running-Stack-and-Send-the-First-Request/page.md): Step-by-step demo to port‑forward a router and Prometheus, send an OpenAI-format completion to a Kubernetes llm-d router, and verify pods, Helm release, and Prometheus metrics. - [Demo Watch One Request Happen](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Tour-the-Running-Stack/Demo-Watch-One-Request-Happen/page.md): Walkthrough showing how to port‑forward to a single model pod, stream a completion, and observe prefill, token decoding, and KV scratchpad metrics in real time. ### Smart Routing, Live - [The Scorers Behind Every Decision](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Smart-Routing-Live/The-Scorers-Behind-Every-Decision/page.md): Explains a Kubernetes router ConfigMap with scorers that prioritize cache reuse and load signals to route requests across pods, optimizing latency and cache distribution - [Demo The Same Prompt Lands on the Same Server](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Smart-Routing-Live/Demo-The-Same-Prompt-Lands-on-the-Same-Server/page.md): Demonstrates how a cluster router and per-pod prefix cache route repeated prompts to the same model server, reusing cached token blocks to reduce latency and CPU - [Demo Cold Requests Spread Out](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Smart-Routing-Live/Demo-Cold-Requests-Spread-Out/page.md): Test demonstrating how a router spreads new concurrent requests across pods, prioritizing cached ownership and resource signals to balance cold requests ### Getting More From the KV Cache - [Precise Prefix Cache Routing](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Getting-More-From-the-KV-Cache/Precise-Prefix-Cache-Routing/page.md): Explains upgrading an approximate router prefix-cache scorer to a precise live reported-index that tracks pod cache events to improve routing accuracy and reduce wasted cache misses. - [Tiered Cache Spill Over to CPU and Disk](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Getting-More-From-the-KV-Cache/Tiered-Cache-Spill-Over-to-CPU-and-Disk/page.md): Explains a tiered KV cache design that demotes model state from GPU to RAM to disk to avoid recomputation and reduce latency for long conversational workloads. - [Demo A Multi Turn Chat Reuses the Cache](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Getting-More-From-the-KV-Cache/Demo-A-Multi-Turn-Chat-Reuses-the-Cache/page.md): Demonstrates prefix caching in multi-turn chats so growing conversation history doesn't increase per-turn processing cost, using a five-turn demo, jq payloads, and Prometheus metrics. ### Serving Bigger Models With llm-d - [Wide Expert Parallelism](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Serving-Bigger-Models-With-llm-d/Wide-Expert-Parallelism/page.md): Explains why Mixture-of-Experts models benefit from expert parallelism, assigning whole experts per GPU, scaling wide and requiring RDMA-capable fabrics for efficient all-to-all communication - [Demo Read the PD well lit Path Against Our Live Cluster](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Serving-Bigger-Models-With-llm-d/Demo-Read-the-PD-well-lit-Path-Against-Our-Live-Cluster/page.md): Explains implementing software-based prefill and decode disaggregation for LLM serving on Kubernetes using role labels, sidecars, EPP scheduling, and NIXL KV transfers. - [What Really Needs GPUs](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Serving-Bigger-Models-With-llm-d/What-Really-Needs-GPUs/page.md): Explains which AI serving behaviors run on CPUs versus which require high‑performance GPU fabric and validated benchmarks for Wide Expert Parallelism ### Running It Day to Day - [Flow Control A Line for Requests](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Running-It-Day-to-Day/Flow-Control-A-Line-for-Requests/page.md): Designing EPP flow control to manage request queues using priority bands, per-tenant fairness and TTLs to prevent noisy neighbors and guide autoscaling. - [Autoscaling on the Right Signals](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Running-It-Day-to-Day/Autoscaling-on-the-Right-Signals/page.md): Autoscaling model servers using inference-aware metrics like queue depth and cache usage, proactive scaling for TTFT SLOs, and a variant autoscaler for cost and latency optimization - [Rollouts With No Downtime](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Running-It-Day-to-Day/Rollouts-With-No-Downtime/page.md): Describes three zero downtime model rollout strategies, including rolling updates, blue green canaries, and LoRA adapter swaps, plus practical Kubernetes readiness and deployment guidance. - [Demo A Rolling Update Under Live Traffic](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Running-It-Day-to-Day/Demo-A-Rolling-Update-Under-Live-Traffic/page.md): Demonstrates zero-downtime rolling updates for model servers under live traffic using a user heartbeat, Kubernetes rollout, and Prometheus readiness monitoring ### Seeing What Is Happening - [The Numbers llm d Reports](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Seeing-What-Is-Happening/The-Numbers-llm-d-Reports/page.md): Guide to organizing LLM serving metrics into pod and fleet sources and four families (startup, streaming, concurrency, cache) with percentile-based monitoring and dashboarding recommendations. - [Demo Read the Grafana Dashboard](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Seeing-What-Is-Happening/Demo-Read-the-Grafana-Dashboard/page.md): Guide to reading a Grafana dashboard for LLM services, interpreting latency, throughput, scheduler and cache metrics, using Prometheus queries and controlled load for troubleshooting - [Demo Run a Small Benchmark](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Seeing-What-Is-Happening/Demo-Run-a-Small-Benchmark/page.md): How to run a small controlled benchmark using GuideLLM to generate synthetic load, measure client and server metrics, and ensure repeatable, trustworthy LLM cluster performance comparisons ### Wrap Up: Agents and Beyond - [Agentic Serving](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Wrap-Up-Agents-and-Beyond/Agentic-Serving/page.md): Optimizing serving infrastructure for agentic model workloads by caching shared context, maintaining program affinity, precise routing, tiered caches, and per-flow fairness. - [Multimodal Serving](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Wrap-Up-Agents-and-Beyond/Multimodal-Serving/page.md): Explains how multimodal models turn images into many tokens, affecting prefill performance, unchanged routing, and benefiting from cached image encodings to reduce compute. - [Demo An Agent Style Multi Turn Workload](https://notes.kodekloud.com/docs/llm-d-A-Deep-Dive/Wrap-Up-Agents-and-Beyond/Demo-An-Agent-Style-Multi-Turn-Workload/page.md): Demonstrates an agent-style multi-turn coding workload, cache growth, and program-level cache effects across a serving fleet with Prometheus metrics.