> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo Cold Requests Spread Out

> Test demonstrating how a router spreads new concurrent requests across pods, prioritizing cached ownership and resource signals to balance cold requests

One promise down, one to go.

The cluster router already honors a strong rule: if a prompt matches saved work it "pulls home" (weight three). We tested that behavior earlier. There is a second, subtler promise: brand-new prompts with no saved work score zero with the prefix scorer, so placement falls to the rest of the system — the load scorers, queues, the scratchpad room, and the newcomer spreader. The newcomer spreader’s job is to prevent fresh cache entries from being concentrated on a single pod.

Our hypothesis: if many identical requests (for example, six copies of the same prompt) hit a single pod, those get pulled home. In contrast, 30 unique requests arriving concurrently should be spread across the pods by the newcomer logic.

What follows is the exact sequence of measurements and commands used to validate that behavior.

## Baseline: check prompt tokens per pod

We query Prometheus for total prompt tokens processed per pod over the last 5 minutes:

```promql theme={null}
sum by(pod) (increase(vllm:prompt_tokens_total[5m]))
```

Before the burst, the metric shows zeros across the three pods:

```text theme={null}
{pod="optimized-baseline-cpu-vllm-decode-5c77c88766-4hmpb"} 0
{pod="optimized-baseline-cpu-vllm-decode-5c77c88766-drbrd"} 0
{pod="optimized-baseline-cpu-vllm-decode-5c77c88766-g7kkb"} 0

Load time: 303ms  Result series: 3
```

## Send a concurrent burst of 30 unique requests

We launch 30 distinct prompts in parallel. Each request contains a counter so every prompt is unique:

```bash theme={null}
for i in $(seq 1 30); do
  curl -s http://localhost:8000/v1/completions \
    -H 'Content-Type: application/json' \
    -d "{\"model\":\"Qwen/Qwen2.5-0.5B-Instruct\",\"prompt\":\"Write one sentence about topic number ${i} in cloud computing.\",\"max_tokens\":20}" \
    -o /dev/null -w "sent ${i}: %{http_code}\n" &
done
wait
```

Because all requests are in flight simultaneously, the completion order is non-sequential and reflects how quickly each pod finished assigned work. Sample stdout from the loop:

```text theme={null}
sent 19: 200
sent 14: 200
sent 4: 200
sent 28: 200
sent 11: 200
sent 6: 200
sent 13: 200
sent 21: 200
sent 22: 200
sent 25: 200
sent 30: 200
sent 24: 200
sent 5: 200
sent 20: 200
sent 15: 200
sent 9: 200
sent 2: 200
sent 26: 200
sent 27: 200
sent 23: 200
sent 29: 200
```

Every line shows HTTP 200 — 30 successful responses — and the scrambled order confirms requests finished as pods completed work, not by a rigid round-robin.

## Post-burst metrics: how work landed across pods

After the burst we re-run the same Prometheus query:

```promql theme={null}
sum by(pod) (increase(vllm:prompt_tokens_total[5m]))
```

Observed distribution:

```text theme={null}
{pod="optimized-baseline-cpu-vllm-decode-5c77c88766-4hmpb"} 254.73684210526315
{pod="optimized-baseline-cpu-vllm-decode-5c77c88766-drbrd"} 52.63157894736842
{pod="optimized-baseline-cpu-vllm-decode-5c77c88766-g7kkb"} 93.68421052631578
```

For clarity, here is a compact summary:

| Pod name | Prompt tokens increased (5m) |
| - | - |
| optimized-baseline-cpu-vllm-decode-5c77c88766-4hmpb | `254.73684210526315` |
| optimized-baseline-cpu-vllm-decode-5c77c88766-drbrd | `52.63157894736842` |
| optimized-baseline-cpu-vllm-decode-5c77c88766-g7kkb | `93.68421052631578` |

Compare that with a scenario where repeated requests were pulled home and concentrated on a single pod. With 30 distinct newcomer requests, work was distributed across all pods. Note that the three numbers are not identical — this is deliberate.

Important nuance: this is not simple round-robin. The router steers newcomers based on live signals (queue depths, memory pressure, recent newcomer placement) so placement is roughly even, not perfectly equal. A blind load balancer would equalize uniformly but would ignore where saved work exists. The router evaluates each request with this single question: where will this request cost the least? If a request matches saved state, pulling home is cheapest. If it’s a newcomer with no saved state, the cheapest location is wherever there is room — hence the spread.

The router infers ownership from observed traffic and uses that signal to prefer pulling prompts "home" when cheap. This heuristic works well but remains a guess. Removing the guess requires a cache that exposes exact ownership metadata to the router so placement can be made deterministically.

## Takeaway

* When requests match existing cached work, the router prefers the pod that already owns that state.
* For cold (new) requests, the newcomer spreader distributes work across pods using live resource signals.
* The result is an intentional, approximate balancing strategy that prioritizes cost (latency and cache reuse) over strict uniformity.

References

* Prometheus: [https://prometheus.io/](https://prometheus.io/)

> **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/a6ab4920-a1d0-4f6a-aa81-fa7b8f8ac0e9/lesson/ded876a6-2484-4191-882e-14b3d311417e)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.