Skip to main content
One promise down, one to go. The cluster router already honors a strong rule: if a prompt matches saved work it “pulls home” (weight three). We tested that behavior earlier. There is a second, subtler promise: brand-new prompts with no saved work score zero with the prefix scorer, so placement falls to the rest of the system — the load scorers, queues, the scratchpad room, and the newcomer spreader. The newcomer spreader’s job is to prevent fresh cache entries from being concentrated on a single pod. Our hypothesis: if many identical requests (for example, six copies of the same prompt) hit a single pod, those get pulled home. In contrast, 30 unique requests arriving concurrently should be spread across the pods by the newcomer logic. What follows is the exact sequence of measurements and commands used to validate that behavior.

​
Baseline: check prompt tokens per pod

We query Prometheus for total prompt tokens processed per pod over the last 5 minutes:
Before the burst, the metric shows zeros across the three pods:

​
Send a concurrent burst of 30 unique requests

We launch 30 distinct prompts in parallel. Each request contains a counter so every prompt is unique:
Because all requests are in flight simultaneously, the completion order is non-sequential and reflects how quickly each pod finished assigned work. Sample stdout from the loop:
Every line shows HTTP 200 — 30 successful responses — and the scrambled order confirms requests finished as pods completed work, not by a rigid round-robin.

​
Post-burst metrics: how work landed across pods

After the burst we re-run the same Prometheus query:
Observed distribution:
For clarity, here is a compact summary: Compare that with a scenario where repeated requests were pulled home and concentrated on a single pod. With 30 distinct newcomer requests, work was distributed across all pods. Note that the three numbers are not identical — this is deliberate. Important nuance: this is not simple round-robin. The router steers newcomers based on live signals (queue depths, memory pressure, recent newcomer placement) so placement is roughly even, not perfectly equal. A blind load balancer would equalize uniformly but would ignore where saved work exists. The router evaluates each request with this single question: where will this request cost the least? If a request matches saved state, pulling home is cheapest. If it’s a newcomer with no saved state, the cheapest location is wherever there is room — hence the spread.
The router infers ownership from observed traffic and uses that signal to prefer pulling prompts “home” when cheap. This heuristic works well but remains a guess. Removing the guess requires a cache that exposes exact ownership metadata to the router so placement can be made deterministically.

​
Takeaway

  • When requests match existing cached work, the router prefers the pod that already owns that state.
  • For cold (new) requests, the newcomer spreader distributes work across pods using live resource signals.
  • The result is an intentional, approximate balancing strategy that prioritizes cost (latency and cache reuse) over strict uniformity.
References

Watch Video