Designing EPP flow control to manage request queues using priority bands, per-tenant fairness and TTLs to prevent noisy neighbors and guide autoscaling.
Earlier, when we opened the router, we sketched three stages and skipped the middle one. The three stages are:
Request handling — read and validate the incoming request.
Flow control — decide whether a request should run now, later, or not at all.
Request scheduling — pick the pod that will run the work.
This article returns to that skipped middle stage. Flow control sits between the front door and the scheduler, and it’s the feature that often determines whether a high-traffic day is manageable or chaotic.
Why flow control mattersImagine a sudden traffic spike: every pod’s batch is full, internal queues are deep, and requests continue to arrive. Some requests must wait. The key question is whether that waiting is orderly or a chaotic pileup. Flow control gives the router a proper waiting room and policies to enforce predictable behavior.
The noisy neighbor problemThe issue is especially acute for language models because requests vary widely in cost. If the router is strict first-come, first-served, a single tenant submitting a 50‑page document can block dozens of short queries that arrived later. Clients then time out and retry, increasing queue depth and amplifying the outage — a self-reinforcing noisy-neighbor failure mode that plain load balancing cannot fix.
Three primary rule categories for a router waiting roomFlow control policies are typically implemented in the EPP (edge/entry-point proxy) where every request already passes. The rules work together to protect critical traffic, enforce fairness, and preserve per-tenant ordering.
Category
What it does
Operational effect
Priority bands
Classify requests into bands (high/medium/low); give higher bands priority and reserved capacity
Keeps important traffic flowing even under load
Fairness across flows
Partition traffic inside a band by flow (tenant or workload) and schedule among flows
Prevents a noisy tenant from monopolizing resources
Order within a flow
Serve requests inside a flow in arrival order (usually FIFO) once the flow has been selected
Preserves request ordering for a single tenant/workload
A fourth operational rule is equally important: every waiting request has a deadline or TTL. If the router can tell a request won’t be served before its deadline, it rejects the request early — returning a fast failure rather than allowing the request to linger and waste resources.
Fast rejection beats slow failure: returning an early, explicit rejection lets clients retry sooner or fall back to cheaper logic. This reduces wasted work and prevents retry storms that deepen queues.
Flow control as an operational instrumentClassic autoscalers use utilization to decide when to add capacity. Utilization is noisy: a GPU at 100% might be working on one enormous prompt or multiplexing many small requests — the metric doesn’t distinguish unmet demand from busy-but-satisfied capacity.Queue depth is different: it’s a direct, observable count of outstanding demand that the fleet has not yet served. Measuring queue depth in one place (the waiting room in the EPP) gives you an honest signal for scaling decisions: when the queue shows unmet demand, the autoscaler can act in a targeted way.
Queue depth also informs a higher-level decision: who should add pods when the fleet is undersized — the router itself, a centralized autoscaler, or the cloud control plane? That is an architecture and policy choice, and queue metrics make the trade-offs visible.
On fairness and queueing theory: search for “fair queuing” and “weighted fair queueing” for algorithms used in multi-tenant systems.
For autoscaler design, look for resources on using queue length or request backlog as a scaling signal.
By implementing a waiting room with priority bands, per-flow fairness, per-flow ordering, and TTL-based early shedding, an EPP can prevent noisy neighbors, reduce retry storms, and provide clean operational signals for autoscaling.
