Explains how a router uses Kubernetes API watches and per-pod metrics scrapes plus Envoy ext_proc to score and select pods for low-latency request routing
The EPP (the routing “brain”) scores every pod and picks the winner for each incoming request. But where do those scores come from?You might expect a central telemetry agent or exotic reporting system, but the truth is simpler: the routing decision is built from two very ordinary Kubernetes primitives — an API watch and a metrics scrape — combined into a compact data layer.Underneath all three stages sits that data layer, and it has exactly two jobs:
Discover which pods exist for a given model.
Observe what those pods are doing right now.
These two pieces together create the cluster’s live view that the EPP uses to score and pick a pod.
There is no static list for you to edit — the cluster is the source of truth. A small custom object called the InferencePool advertises which pods are serving a particular model; a controller watches the Kubernetes API server and keeps that list current. The data layer therefore maintains the current set of pod endpoints, for example:
10.0.4.1210.0.4.1310.0.4.19
The second job is to know what those pods are doing at any instant. Each serving pod exposes a lightweight metrics endpoint; runtimes such as vLLM already publish these metrics. A typical pod reports an instantaneous snapshot like this:
GET /metricsvllm:num_requests_waiting 3vllm:num_requests_running 12vllm:gpu_cache_usage_perc 0.78
Your existing monitoring may already be scraping this same endpoint.
The EPP continually reads those metrics from every pod in the pool. In practice this is just two simple loops working together:
A controller watches the Kubernetes API server to discover pod endpoints (the InferencePool).
A scraper loop polls each pod’s /metrics endpoint to capture its instantaneous state.
That API-watch plus metrics-scrape is the entire live view of the fleet — two basic Kubernetes primitives solving a load-balancing problem that plain L4/L7 load balancers cannot resolve by themselves.
One piece remains: the gateway that accepts client connections. That gateway typically runs an L7 proxy — commonly Envoy — which actually holds the client connection. The EPP runs in a separate pod, so the proxy and the EPP need a way to exchange routing decisions without shunting the request body through the EPP.They use Envoy’s ext_proc (external processing) hook. Ext_proc is a built-in Envoy feature that lets the proxy pause a request, call out to an external service, and use that service’s response to influence the proxy’s next action. The router simply plugs its decision into this existing hook — no custom proxy protocol is required.
Important: the EPP is not a hop in your request’s data path. The sequence is:
The L7 proxy (Envoy) receives the client request and pauses processing.
Envoy sends a small routing query (metadata about the request) to the EPP via ext_proc.
The EPP returns a pod name or endpoint.
Envoy forwards the original request directly to the chosen pod.
The pod response streams back to the client through Envoy.
So tokens never travel through the EPP — only a tiny routing query does.
The EPP only receives a tiny routing query (metadata about the request) and returns an endpoint. It never handles the request body or response stream, which keeps the EPP lightweight and low-latency.
This architecture explains why the routing brain can stay small: it never moves gigabytes of tokens — it only answers a quick question. Every scoring decision costs a single fast RPC (the ext_proc call) rather than adding an extra hop for each token in the response.But keep in mind what the EPP actually reads: instantaneous snapshots of each pod. The metrics look like this:
GET /metricsvllm:num_requests_waiting 3vllm:num_requests_running 12vllm:gpu_cache_usage_perc 0.78
Those three numbers are point-in-time signals: queue depth now, running work now, and cache usage now. A pod that looks ideal at this instant may soon be occupied by a long-running job that arrived a millisecond earlier. The EPP cannot reliably foresee long requests from instantaneous metrics alone.So — what if the router could predict what’s about to happen, rather than only seeing what’s happening right now?
Next steps usually explore adding prediction signals (request size estimates, model hints, historical patterns) or lightweight admission control to improve routing accuracy. But the foundational idea remains: the EPP combines a Kubernetes API watch and a per-pod metrics scrape to form a live, low-latency routing brain.Summary table
Data-layer job
Mechanism
Purpose
Discover pods
Watch the Kubernetes API server (InferencePool)
Keep an up-to-date list of pod endpoints
Observe pod state
Scrape each pod’s /metrics endpoint (vLLM and similar runtimes)
Provide instantaneous signals for routing decisions
