Demo Tour the Running Stack and Send the First Request

Time to collect on a promise: we’ll make a real OpenAI-style request against the cluster you installed. This page walks through the entire flow end-to-end: port-forwarding the router and Prometheus, confirming the Helm release and pods, sending a completion request, and verifying the router and Prometheus metrics. Before you begin: everything in this demo lives in a single Kubernetes namespace. Set it once per terminal:

export NAMESPACE=llm-d

Overview of what we’ll do:

Port-forward the router (the router’s front door) and Prometheus in two separate terminals
Confirm the Helm release and inspect running pods
Send an OpenAI-format completion request to the router
Verify the request in Prometheus metrics

Open two terminal windows. In one run the port-forward for the router; in the other run the port-forward for Prometheus. Leave both running while you follow this lesson.

Example port-forward commands (run each in a separate terminal):

# Terminal A: front door (router service)
kubectl port-forward service/quickstart-epp 8000:8000 -n ${NAMESPACE}

# Terminal B: Prometheus
kubectl port-forward service/prometheus 9090:9090 -n ${NAMESPACE}

(If your Prometheus service has a different name, substitute it for prometheus.) When both port-forwards run, the router will be reachable at http://localhost:8000 and Prometheus at http://localhost:9090. Quick reference — services and local ports:

Service	Purpose	Local URL
Router front door (`quickstart-epp`)	Receives OpenAI-format requests	`http://localhost:8000`
Prometheus	Monitoring and metrics	`http://localhost:9090`

Inspect Helm and the release installed in the namespace:

helm list -n ${NAMESPACE}

Representative output:

NAME        NAMESPACE  REVISION  UPDATED                                 STATUS    CHART                                   APP VERSION
quickstart  llm-d      1         2026-07-30 14:07:05 -0400 EDT          deployed  llm-d-router-standalone-v0.9.0        v0.9.0

The Helm release name is quickstart (this chart installed the router).
The router’s service you port-forwarded is named quickstart-epp (release + job label).

Inspect the pods in the namespace:

kubectl get pods -n ${NAMESPACE} -o wide

Representative output (cleaned):

NAME                                                          READY  STATUS   RESTARTS  AGE    IP               NODE                 NOMINATED NODE  READINESS GATES
optimized-baseline-cpu-vllm-decode-5c77c88766-4hmpb           1/1    Running  0         2d17h  10.244.223.197   demo-dev-worker-2    <none>          <none>
optimized-baseline-cpu-vllm-decode-5c77c88766-drbrd           1/1    Running  0         2d17h  10.244.108.134   demo-dev-worker-1    <none>          <none>
optimized-baseline-cpu-vllm-decode-5c77c88766-g7kkb           1/1    Running  0         2d17h  10.244.12.140    demo-dev-worker-3    <none>          <none>
quickstart-epp-5f6cbf8c4-tcl7q                                2/2    Running  0         14d    10.244.223.194   demo-dev-worker-2    <none>          <none>

What you should notice:

There are three model server pods (copies of Qwen) and one router pod.
This cluster is running in standalone mode: there is no Gateway pod or Gateway API installed. Instead, Envoy (the proxy) is colocated inside the router pod next to the EPP (endpoint picker), and the router exposes that Envoy via the quickstart-epp Service. In other words: proxy + picker in the same pod, exposed as a standard K8s Service.
The router watches a small llm-d resource whose selector matches labels on the model pods. That selector builds the inference pool dynamically — no manual server inventory is required.

Many models on Hugging Face require an account and access token to download. The quickstart Helm chart provides a place to store such tokens as a Kubernetes Secret. This Quickstart used a public model (Qwen) so no secret is present. If you run gated models on your cluster, create the Kubernetes Secret but never print its value to the terminal.

Now send a real OpenAI-format completion request to the router (via the port-forward):

curl -s http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","prompt":"How are you today?"}' | jq

Example JSON response (formatted by jq):

{
  "id": "cmpl-4697d272-adc1-4c96-832d-cddcef707592",
  "object": "text_completion",
  "created": 1786714656,
  "model": "Qwen/Qwen2.5-0.5B-Instruct",
  "choices": [
    {
      "index": 0,
      "text": " I'm a machine learning model, so my responses are based on algorithms and data",
      "logprobs": null,
      "finish_reason": "length",
      "stop_reason": null,
      "token_ids": null,
      "prompt_logprobs": null,
      "prompt_token_ids": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "prompt_tokens": 5,
    "total_tokens": 21,
    "completion_tokens": 16,
    "prompt_tokens_details": null
  },
  "kv_transfer_params": null
}

What happened end-to-end:

Envoy (inside the router pod) accepted the HTTP request at localhost:8000.
The EPP evaluated and scored the three available model server endpoints, picked one, and forwarded the request.
A model server produced the completion tokens and returned the response to the router, which returned it to your curl client.

Prometheus has been collecting metrics the whole time. With the Prometheus port-forward open on localhost:9090, query the router’s view of the pool. The metric llm_d_epp_ready_endpoints reports how many ready endpoints the router knows about. A representative metric scrape immediately after the curl:

llm_d_epp_ready_endpoints

llm_d_epp_ready_endpoints{container="epp", endpoint="http-metrics", instance="10.244.223.194:9090", job="quickstart-epp", name="quickstart", namespace="llm-d", pod="quickstart-epp-5f6cbf8c4-tcl7q", service="quickstart-epp"} 3

The router agrees with kubectl get pods — there are three ready model servers behind the frontend.
If you run a Prometheus query for the request counter (e.g., the request-handling counter for the EPP) after the curl, you should see that counter increment by one.

This tour checks the full stack: Helm release, pods, standalone router (no Gateway), inference pool, the front door, and a real completion with a verifiable Prometheus trail. Next: we’ll slow a single request down to demonstrate pause, streaming, and scratchpad behavior. References and further reading:

From ChatGPT to a Model Server

Inside a Single Request

From One Server to a Fleet

Meet llm-d

Tour the Running Stack

Smart Routing, Live

Getting More From the KV Cache

Serving Bigger Models With llm-d

Running It Day to Day

Seeing What Is Happening

Wrap Up: Agents and Beyond

Demo Tour the Running Stack and Send the First Request

Watch Video