Skip to main content
In production, you don’t run a single server — you run a fleet. Each machine is typically a GPU-backed server that holds its own copy of the model and serves a subset of users. The catch: the KV cache (the per-conversation key/value state that makes continuing a chat fast) lives only in the memory of the single server that produced it. It is not automatically shared across the fleet. So when the next message arrives, which server handles it makes a huge difference. If it lands on the server already holding your conversation’s KV cache, you get an instant reply. If it lands on any other server, that server must re-read the entire conversation and rebuild the cache, and you wait through the long processing pause again.
A diagram titled "Which Server?" showing a user node branching to five GPU server boxes labeled Server 1–5, each containing a "Model" module. The image illustrates selecting or routing requests among multiple GPU-backed servers (one option marked "Instant").
Because the KV cache is colocated with a specific server, servers in the fleet are not interchangeable. Traditional load balancing treats them as if they were — and that mismatch is the root cause of poor performance for LLM workloads. A standard remedy in web infrastructure is to put a load balancer in front of the servers and spread requests evenly. Round-robin, least-connections, and similar algorithms have balanced HTTP traffic for decades. But for large language models, the default approach is the wrong tool for the job.
A diagram titled "A Load Balancer" showing a central load balancer distributing traffic to five GPU-equipped servers (Server 1–Server 5), each hosting a model. Icons for NGINX and load-balancing are shown above the balancer.
Consider a simple sequence:
  1. You send the first message. The balancer routes it to server 2, which builds the KV cache for your session.
  2. You send a second message. The balancer (e.g., round-robin) routes it to server 5. Server 5 has no cache for your session and must re-process the whole conversation.
A plain load balancer discards already-computed state by treating servers as interchangeable. For LLM serving, that wastes GPU cycles and increases latency. Other “fixes” like session affinity (sticky sessions) can reduce cache misses, but they don’t solve request heterogeneity — another decisive factor. Requests vary widely in compute cost and latency. The load balancer sees only the transport layer (TCP/HTTP) and cannot distinguish a tiny “hello” from a CPU/GPU-heavy “summarize 50 pages” request.
A dark-themed infographic titled "No Two Requests Alike" showing a "Load Balancer" control in the center. To the left is a small "hello" request and to the right a stack labeled "50 pages," with a not-equal symbol between them.
That heavy request might be routed to a server already streaming answers for many users. The heavy job can monopolize the GPU, causing all concurrent sessions on that server to stutter.
A stylized diagram titled "One Huge Request" showing a busy GPU server on the left with stacked server icons and blue compute blocks. Green request nodes flow from the server toward a grid of user icons on the right.
A plain load balancer lacks the visibility required to place work correctly. It does not know:
  • Which KV caches are resident on which servers.
  • How full each server’s memory is.
  • The per-server queue lengths or current compute utilization.
  • The expected cost or latency of an incoming request.
Without that information, the balancer cannot optimize placement and ends up either forcing recomputation or creating resource contention.
A diagram titled "The Whole Problem" showing a user request (with a question mark) being routed to five GPU-backed servers (Server 1–5), each running a model. A "Smarter routing" label highlights the routing decision between the user and the servers.
To summarize the mismatch: What we need is a router that understands session locality and request cost and makes placement decisions accordingly. One open-source project built for this purpose is llm-d. It sits in front of the GPU fleet and routes each request using the data required to make intelligent placement decisions: which server holds the session cache, how busy each server is, and how costly the incoming request will be.
A stylized diagram titled "Our Main Character" shows a user connected to a central open-source router that "routes intelligently." The router links to five GPU servers labeled Server 1–Server 5, each hosting a model.
What to look for in an LLM-aware router:
  • Session affinity based on cache location, not just source IP.
  • Per-request cost estimation (e.g., token counts, expected generation length).
  • Real-time server telemetry (memory usage, queue lengths, GPU utilization).
  • Graceful handling of heavy jobs (queuing, preemption, or directed placement).
  • Metrics and observability for cache hit rates and tail latency.
Next: we’ll explain how to move from blind load balancing to a router that preserves locality and respects request cost, so you can utilize your GPU fleet efficiently and keep latency low.
Key takeaway: traditional load balancers assume interchangeable servers and uniform requests. LLM serving requires routing that is aware of session locality (the KV cache) and request cost to avoid recomputation and resource contention.

Watch Video