Skip to main content
Earlier we covered the vocabulary. In this lesson we walk the wall of charts an on-call engineer actually lives on — the Grafana dashboard built from the exact numbers you just learned, collected from our cluster. Key things to notice up front:
  • Dashboards auto-update — the charts are live and moving.
  • A useful dashboard is not wallpaper: in this demo two systems are running behind the scenes:
    • Grafana itself (served on port 3000 in this setup, typically reachable via a port-forward).
    • A steady load generator that keeps traffic flowing through the router.
The workload here is modest but continuous: about one request every couple of seconds, each request producing a few dozen tokens. On our relatively slow CPUs each response takes time to generate, so with a new request every two seconds there are usually several in-flight requests and occasional short queues. That creates exactly the signal shapes you want to observe in a healthy fleet. Grafana is a visualization layer — it draws, it does not collect. In this dashboard, panels run PromQL queries against Prometheus at a periodic refresh interval; Grafana then visualizes the returned time series.
Grafana panels execute PromQL queries against Prometheus. The dashboard only visualizes the metrics Prometheus stores — it does not change what is measured.
A representative PromQL query used on this dashboard (computing the 99th percentile of request latency across histogram buckets) looks like:
Nothing new is being measured here — these are the same metrics you’ve seen before; the dashboard simply dresses them for human consumption. Below we step through the key panels, what to read from each, and how to prioritize your scan when troubleshooting.

​
Key dashboard panels and how to read them

Time-to-first-token latency
  • This panel plots percentile curves over time. The P50 (median) is usually stable and gives you the typical user experience.
  • The P99 reveals tail behavior — the slowest 1% of requests — and often spikes above the P50. Investigate P99 when you see user-visible latency outliers.
A dark-themed Grafana dashboard titled "llm-d vLLM Overview" showing Prometheus metrics with time-range controls and dropdown filters. Visible panels include line charts for "Time To First Token Latency" and spike graphs for "Cache Utilization" and other request metrics.
Token throughput
  • Shows two series types: prompt tokens read (incoming) and generated tokens written (outgoing).
  • Multiple pods produce multiple series; in this demo there are three pods and you’ll see three lines per metric. Use these lines to verify reading and streaming behavior across replicas.
Scheduler state
  • Panels show requests in running vs. waiting states. These curves reveal concurrency and queuing behavior.
  • If arrivals bunch up the waiting line rises; when processing catches up the queue drains. This pattern is a direct signal of load vs. capacity.
Cache utilization
  • Scratchpad/cache usage is usually plotted one series per pod. Divergence between pod lines can indicate uneven load or stateful hotspots.
  • Watch for sustained growth in cache usage on a single pod — it may point to state accumulation or memory leaks.
Summary table — what to look for When you step back and scan the entire dashboard, one important observation stands out: every panel in this CPU-based setup is ultimately driven by CPU-bound metrics and the absolute values are relatively small and slow. The absolute numbers change if you move to GPUs, but the shapes, correlations, and the order in which you scan the panels remain the same. Good dashboard-reading habits translate directly to larger fleets.
A Grafana dashboard titled "llm-d vLLM Overview" showing Prometheus metrics with line charts for E2E Request Latency and Token Throughput over the last 15 minutes. The UI includes filters (datasource, model_name, namespace), chart legends, and dashboard controls.

​
From observation to verification: generate controlled load

This dashboard shows how the fleet behaves under whatever traffic happens to exist. To compare configurations or prove system performance you must generate controlled traffic:
  • Use a reproducible load generator that can fix arrival patterns (constant rate, bursts, or specific concurrency).
  • Measure the same panels consistently across experiments (same time window, same filters).
  • Keep a baseline run and compare percentiles (P50, P95, P99), throughput, and queue lengths.
For trustworthy comparisons, create controlled load and compare identical dashboard panels across runs. Visual inspection plus percentile metrics (P50/P95/P99) provides the best quick signal for performance regressions.

​
References and further reading

This walkthrough should help you read the wall of charts: glance at pods, then streams, then tail latency and the scratchpad. Those reading habits are what turn a dashboard into an action plan when incidents occur.

Watch Video