Skip to main content
The cluster dashboard shows how the fleet behaves under whatever traffic happens to arrive, but the engineering questions that drive decisions are more specific:
  • Would four pods outperform three?
  • Did that configuration change actually help?
  • Can this setup guarantee P99 under 2 seconds?
To answer questions like these you need a controlled, repeatable experiment — a benchmark. A benchmark applies the same synthetic load twice (or more), isolates the variable you care about, and produces comparable numbers. All published numbers in this article were produced that way. Before running the benchmark two things were done:
  1. The background loop that had been running was stopped. A benchmark must own the traffic; if other traffic runs concurrently the results are meaningless.
  2. The benchmark run itself was executed; it completed before this article was started. A run typically takes a couple of minutes, and watching a progress bar is not the point — capturing repeatable metrics is.
Stop any background or synthetic traffic before benchmarking. A clean test environment is essential to produce repeatable, trustworthy measurements.
Tool used for the run: GuideLLM — a client-side load generator (installed via pip). Its job is to act like a disciplined crowd: send synthetic requests shaped and paced by you, measure every latency from the client side, and emit the percentiles and throughput that matter for user experience. Here is the exact command that produced the run. Read the flags like a sentence because they encode the whole experiment design:
What each flag means:
  • --backend kind=openai_http,target=http://localhost:8000 — target the router’s front door with the OpenAI HTTP format.
  • --data kind=synthetic_text,prompt_tokens=128,output_tokens=64 — make every request synthetic and identical: 128 prompt tokens in, 64 out.
  • --profile kind=constant,rate=2 — use a metronome: a constant rate of 2 requests per second.
  • --constraint kind=max_duration,seconds=120 — set a curfew: stop the run after 120 seconds.
A metronome and a curfew: the load generator sends a steady two requests per second for two minutes. While the run executes the fleet experiences its busiest two minutes: requests can arrive faster than slow pods can complete responses, concurrency climbs, and queues form. Below is an excerpt of GuideLLM’s console output for that run (shortened for clarity):
When the run finishes, GuideLLM prints the client-side, end-to-end measurements you care about: requests per second actually served, time-to-first-token (median and P99), and percentile latency numbers. Those numbers reflect real user experience because they are measured from the client side. Server-side throughput is also reported. Here is a readable server-side summary for all requests: ✓ Benchmarking complete, generated 1 benchmark(s)
… json : /Users/cbshort/benchmarks.json
… csv : /Users/cbshort/benchmarks.csv What to watch on a saturated fleet
  • Time-to-first-token (TTFT) — usually reported at median and P99 — indicates how quickly the first byte of a response begins.
  • Inter-token latency (the stream gap between tokens, sometimes called TPOT) and output tokens/sec capture streaming behavior and overall generation speed under the specific load.
  • Concurrency, queue lengths, and server-side throughput show how much work is in flight and how well the fleet is keeping up.
Repeat the exact same run after changing only the variable you care about (for example, pod count or CPU limits) and you have an honest comparison — that is the entire point of benchmarking. One last habit distinguishes someone who trusts numbers: never rely on a single witness. Inspect the benchmark’s two-minute window in Grafana and you will see a consistent fingerprint across dashboards: a two-minute swell rising and falling on panel after panel — more work running, more work waiting, longer pauses. When the client-side tool, server telemetry, and Grafana dashboards all agree, you can confidently present the result.
A screenshot of a Grafana dashboard titled "llm-d vLLM Overview" showing Prometheus time-series panels and filter controls. The panels display line charts for metrics like requests prefill/decode time and max generation tokens over a short time interval.
Best practice checklist for small benchmarks:
  • Stop background traffic.
  • Define synthetic request shape and rate.
  • Use a fixed-duration or fixed-count constraint.
  • Collect client-side percentiles, server-side throughput, and Grafana traces.
  • Repeat runs to confirm differences are real.
These CPU numbers are small by design, but the method scales: a controlled load, a percentile table, a second witness, and repeatability. The same tooling and procedure produced the published GPU results at larger scales — the methodology holds even when cluster size grows.

Watch Video