Autoscaling model servers using inference-aware metrics like queue depth and cache usage, proactive scaling for TTFT SLOs, and a variant autoscaler for cost and latency optimization
Every Kubernetes team has the reflex: load rises, the autoscaler reads a utilization number, and pods get added. That pattern works for many stateless web services, but it often fails for model servers. This guide explains why CPU-based autoscaling misfires for inference workloads and how llm-d addresses the two core problems: the signal and the timing.
Start with the signalThe classic autoscaler watches processor utilization. As explained above, that single number is often misleading for inference workloads.
Why CPU% is a poor proxy for inference demand
A server at “full tilt” may be serving one very large prompt or many small multiplexed prompts — both can produce similar CPU utilization but very different user impact.
Modern decoding (token generation) is frequently memory- and memory-bandwidth-bound: cache pressure and KV cache hits dominate latency more than raw CPU cycles.
In short, the meter being watched (CPU%) is often not attached to the scarce resource that determines user experience.
Use direct inference signals insteadllm-d replaces the CPU proxy with direct signals: queue depth, requests in flight, and KV cache pressure — metrics that reflect demand and available capacity directly. The router (EPP) already exposes these metrics so the autoscaler can act on truthful data instead of guesses:
GET /metricsvllm:num_requests_waiting 3vllm:num_requests_running 12vllm:gpu_cache_usage_perc 0.78
Key metrics and their meaning
Metric
Meaning
Why it matters
vllm:num_requests_waiting
Number of requests queued
Indicates demand building before latency spikes
vllm:num_requests_running
Concurrent in-flight requests
Shows current active load and capacity usage
vllm:gpu_cache_usage_perc
Fraction of GPU cache used
Captures memory pressure and cache saturation
From a monitoring perspective, these metrics let you reason about both demand and the true bottlenecks (e.g., cache saturation) rather than inferring them from CPU alone.Timing: why reactive scaling is often too slowStateless web pods can serve traffic within seconds after startup. Model servers are different: they must load model weights and warm caches before they can generate tokens efficiently. A reactive scale-up that waits for CPU% or latency to spike typically brings new pods online only after users are already suffering.
Reacting too late means answering the spike only after the pain occurred.
Be proactive: scale on early demand indicatorsQueue depth typically rises before latency does. When requests begin to queue, the system has time to load models and warm caches while user pain is still theoretical. That allows new capacity to become useful before the Time To First Token (TTFT) SLO breaks.The autoscaling signals should therefore be SLO-aware: the objective is not a tidy CPU graph but to add capacity before the TTFT SLO degrades, and to remove capacity only when the fleet is genuinely idle.
Scaling in: preserve working setsScaling in requires extra caution. Evicting a pod drops its scratchpads and any in-memory conversation state. Conversations whose working set lived on that pod may need a full reread of context unless a tiered cache preserved the blocks on cheaper storage. Scale-in can reduce infra costs but, if done carelessly, imposes a latency tax on users when conversations are resumed.
Careful: evicting pods without preserving cached blocks forces re-reads of model context when those conversations are resumed, increasing latency for users.
Practical implementationsShort path (works for most deployments)
The HPA scales as usual — but now on honest, inference-aware signals.
Advanced path: Workload Variant AutoscalerFor larger deployments, llm-d provides a second, smarter layer: the Workload Variant Autoscaler.
A “variant” is the same model served with a different configuration or hardware profile (different GPU types, batch sizes, or memory allocations).
Each variant has its own cost and latency profile.
The Variant Autoscaler watches the same truthful signals — cache utilization and queue depth — plus any budgets or latency targets you set.
It “shops around” for capacity: when additional resources are needed, it prefers the cheapest variant that still meets the latency target; when you can shrink, it recommends removing the most expensive variants first.
Separation of concernsThe Variant Autoscaler does not directly perform the HPA action. Instead:
It publishes recommendations as metrics.
The ordinary HPA consumes those metrics and executes the scaling decision.
This clear division keeps the control plane you trust (HPA) in charge of API-level scaling actions while an inference-aware layer provides financially and latencysensitive advice.Operational considerations: rolling changes and orchestrationGrowing and shrinking a fleet under live traffic — new versions, new models, and new adapters — requires careful rollout practices. Smooth upgrades need orchestration that avoids disrupting in-flight requests and preserves cached state when possible.
Tip: Start by replacing CPU-based HPA triggers with direct router metrics (num_requests_waiting, num_requests_running, gpu_cache_usage_perc) and tune thresholds against your TTFT SLO. Introduce variant-level recommendations once those signals are stable.
