Skip to main content
We’ve built the complete picture — from a single GPU to llm-d routing across a fleet and coordinating the largest models when they’re spread across many machines. That’s the full capability. Now let’s turn that design into a working deployment. All the concepts we’ve covered map to concrete knobs and tradeoffs you can tune: the router balances saved work in the KV cache against current load, you can split servers that read prompts from servers that write answers, choose how many of each pool to run, decide how many GPUs a single model replica spans, and set how many requests each server batches together. Every one of those settings moves the needle on latency, throughput, and cost. The optimal configuration depends on your model architecture, hardware, and traffic patterns. Manually exploring every combination would be slow and error-prone. Instead, llm-d provides tested starting points — called well-lit paths — so you begin with sensible, measured defaults and iterate from there.
A neon-styled infographic titled "Real Knobs" showing four numbered panels with dials and icons that represent tuning options: cache vs load, split pools (read/write), GPUs per copy, and batch size. Each panel has a circular knob graphic and small illustrative icons.
Key knobs and what they affect: Three well-lit deployment paths cover the common production patterns — each maps the same router to a different fleet shape.
  • Path 1 — Optimized baseline
    A single, homogeneous pool of servers with cache-aware routing enabled. The router prefers servers that already hold relevant KV cache for an incoming request; this reduces repeated compute and improves both throughput and tail latency. On identical hardware, enabling cache-aware routing produced the ~3× improvement referenced earlier.
A dark-themed diagram titled "Path 1 – The Optimized Baseline" showing a router on the left connected to one pool of four pods labeled pod 0 through pod 3. A highlighted "Cache-aware" toggle is enabled and two comparison bars below show "Spread evenly" versus "Cache-aware" performance.
  • Path 2 — Split the work (prefill + decode)
    Divide your fleet into a prefill pool that reads and processes long prompts (building the KV cache) and a decode pool that performs autoregressive decoding to produce answers. The KV cache flows from prefill servers to decoders so decoders avoid redoing expensive prefix computation. Size each pool to match your traffic shape — this pattern shines when prompts are long or requests are latency-critical.
A dark-themed diagram titled "Path 2 – Split the Work" showing a router labeled "llm-d" directing prompts to a "Prefill Pool – Reads Prompts" (pre0, pre1) and sending work to a "Decode Pool – Writes Answers" (dec0, dec1, dec2), with an arrow labeled "Saved work" between them. The slide includes copyright KodeKloud.
  • Path 3 — Model sharding for very large models
    When a model is too big for a single device, one logical serving endpoint is sliced across many GPUs (model parallelism). A single replica spans a group of devices that cooperate to act as one server, enabling inference for the largest models.
Notice what stays constant across all three paths: the router. Its API and routing logic never change — only the shape of the fleet behind it does. That consistency lets you evolve your deployment incrementally without rewriting your client-side integration.
A presentation slide titled "Same Router, Different Fleet" shows three identical router modules labeled "llm-d" under a green "Never changes" bar, each with an arrow pointing to different groups of service/container icons. The graphic illustrates the same router being used across multiple deployment fleets.
Most teams start with the optimized baseline (Path 1) because it delivers a large win with minimal architectural complexity. From that stable foundation you can iterate:
  • If you see many long, repeated prefixes or heavy prompt processing, split the fleet into prefill and decode pools.
  • If the model requires more memory or compute than a single GPU can provide, move to sharded replicas.
  • Adjust router cache weighting, batch sizes, and pool sizes to trade off latency vs throughput and cost.
Start with the optimized baseline (cache-aware routing). It delivers a strong, low-risk improvement and gives you a stable platform to evaluate whether you need to split pools or adopt model sharding.
Next: standing up llm-d on a cluster. We’ll deploy an instance, inspect the resources and services it creates, and use those endpoints to send production-style requests and measure latency and throughput.
A stylized architecture diagram titled "Next – Standing It Up" showing a cluster with connected "Gateway" and "Scheduler" components above three pod boxes labeled "pod 0", "pod 1", and "pod 2." The Gateway is purple and the Scheduler is blue-green.
Further reading and references
  • llm-d routing and cache-aware strategies — see your llm-d router documentation for API details and routing parameters.
  • Best practices for model sharding and device placement — follow your framework’s model parallelism guides for optimal memory and performance planning.

Watch Video