Skip to main content
This lesson explains the infrastructure that powers modern AI — the GPUs, servers, and orchestration systems behind every product you use. If you work as an SRE, systems administrator, or DevOps engineer, this material is directly relevant to the systems you’ll design, deploy, and operate. Companies such as Amazon, Google, Microsoft, and Meta are collectively spending more than $700 billion on AI infrastructure this year — roughly four times what they spent four years ago, before ChatGPT launched. Spending grew nearly 80% in the past year alone, and McKinsey projects about $7 trillion of investment in data center capacity by 2030. All that capacity must be designed, deployed, and kept running — and engineers who can do that are among the most sought-after in the industry.
An infographic titled "Big Tech AI Spending 2022–2025" showing a colorful stacked area chart of AI capex for companies like Microsoft, Amazon, Alphabet, Meta, and Oracle, with callouts reading "$700B+ this year" and "4x since ChatGPT launched." A small circular thumbnail of a person appears in the bottom-right.
Why study this? Because understanding the full stack — from GPU hardware to distributed orchestration — lets you optimize cost, reduce latency, and scale safely as demand grows. Below is a concise roadmap of what we’ll cover and why each topic matters.
This material assumes you can follow how a request flows through a system. You do not need a machine learning research background — familiarity with systems concepts (networking, caching, and servers) is sufficient.
So, what actually happens when you type a prompt into ChatGPT and press Enter? The end-to-end system is layered and stateful:
  • A request enters the front-end API and is routed to a model server that has an instance of the model or relevant cached state.
  • The model server performs token-by-token generation; each new token usually requires attention over past tokens, which is why efficient key/value caching matters.
  • To maximize GPU utilization, servers may batch requests or pack multiple users onto a single GPU, balancing latency and throughput.
  • For models that don’t fit on one GPU, model parallelism splits computation across multiple cards while keeping communication overhead manageable.
  • Conventional stateless load balancing breaks down because of cache locality and model partitioning — the orchestrator needs to be cache-aware and model-aware when routing requests.
We’ll explore all of those layers in detail, starting with a single GPU and evolving that into a full model server. You’ll see concrete diagrams and code-level explanations for how caches are stored and moved, how to shard weights across GPUs, and how to safely scale a fleet of GPU-backed servers.
A presentation slide titled "What We'll Cover" listing topics about GPUs, model servers, text generation, caching, multi-user GPU use, serving giant models, and load balancing. The slide shows the "llm-d" logo, simple diagrams, and a small webcam thumbnail of the speaker.
You’ll also learn how an open-source orchestration project — maintained by contributors from companies like Red Hat, Google, IBM, and NVIDIA — fits into the stack. This orchestrator routes requests to suitable servers, migrates caches when necessary, enforces fair GPU allocation across users, and scales resources down when traffic drops. My name is Mumshad Mannambeth, and I’ll walk you through the material with practical diagrams, real-world trade-offs, and deployment patterns you can apply to production systems. Links and references
  • Kubernetes — Concepts and Architecture
  • McKinsey analysis on data center investment trends (search for McKinsey data center investment 2030)
  • NVIDIA documentation for GPU architectures and multi-GPU programming
  • [Open-source orchestration projects and contributors — Red Hat, Google, IBM, NVIDIA] (search relevant project repositories and docs)

Watch Video