> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo Tour the Running Stack and Send the First Request

> Step-by-step demo to port‑forward a router and Prometheus, send an OpenAI-format completion to a Kubernetes llm-d router, and verify pods, Helm release, and Prometheus metrics.

Time to collect on a promise: we'll make a real OpenAI-style request against the cluster you installed. This page walks through the entire flow end-to-end: port-forwarding the router and Prometheus, confirming the Helm release and pods, sending a completion request, and verifying the router and Prometheus metrics.

Before you begin: everything in this demo lives in a single Kubernetes namespace. Set it once per terminal:

```bash theme={null}
export NAMESPACE=llm-d
```

Overview of what we'll do:

* Port-forward the router (the router's front door) and Prometheus in two separate terminals
* Confirm the Helm release and inspect running pods
* Send an OpenAI-format completion request to the router
* Verify the request in Prometheus metrics

Open two terminal windows. In one run the port-forward for the router; in the other run the port-forward for Prometheus. Leave both running while you follow this lesson.

Example port-forward commands (run each in a separate terminal):

```bash theme={null}
# Terminal A: front door (router service)
kubectl port-forward service/quickstart-epp 8000:8000 -n ${NAMESPACE}

# Terminal B: Prometheus
kubectl port-forward service/prometheus 9090:9090 -n ${NAMESPACE}
```

(If your Prometheus service has a different name, substitute it for `prometheus`.)

When both port-forwards run, the router will be reachable at `http://localhost:8000` and Prometheus at `http://localhost:9090`.

Quick reference — services and local ports:

| Service | Purpose | Local URL |
| - | - | - |
| Router front door (`quickstart-epp`) | Receives OpenAI-format requests | `http://localhost:8000` |
| Prometheus | Monitoring and metrics | `http://localhost:9090` |

Inspect Helm and the release installed in the namespace:

```bash theme={null}
helm list -n ${NAMESPACE}
```

Representative output:

```text theme={null}
NAME        NAMESPACE  REVISION  UPDATED                                 STATUS    CHART                                   APP VERSION
quickstart  llm-d      1         2026-07-30 14:07:05 -0400 EDT          deployed  llm-d-router-standalone-v0.9.0        v0.9.0
```

* The Helm release name is `quickstart` (this chart installed the router).
* The router's service you port-forwarded is named `quickstart-epp` (release + job label).

Inspect the pods in the namespace:

```bash theme={null}
kubectl get pods -n ${NAMESPACE} -o wide
```

Representative output (cleaned):

```text theme={null}
NAME                                                          READY  STATUS   RESTARTS  AGE    IP               NODE                 NOMINATED NODE  READINESS GATES
optimized-baseline-cpu-vllm-decode-5c77c88766-4hmpb           1/1    Running  0         2d17h  10.244.223.197   demo-dev-worker-2    <none>          <none>
optimized-baseline-cpu-vllm-decode-5c77c88766-drbrd           1/1    Running  0         2d17h  10.244.108.134   demo-dev-worker-1    <none>          <none>
optimized-baseline-cpu-vllm-decode-5c77c88766-g7kkb           1/1    Running  0         2d17h  10.244.12.140    demo-dev-worker-3    <none>          <none>
quickstart-epp-5f6cbf8c4-tcl7q                                2/2    Running  0         14d    10.244.223.194   demo-dev-worker-2    <none>          <none>
```

What you should notice:

* There are three model server pods (copies of Qwen) and one router pod.
* This cluster is running in standalone mode: there is no Gateway pod or Gateway API installed. Instead, Envoy (the proxy) is colocated inside the router pod next to the EPP (endpoint picker), and the router exposes that Envoy via the `quickstart-epp` Service. In other words: proxy + picker in the same pod, exposed as a standard K8s Service.
* The router watches a small llm-d resource whose selector matches labels on the model pods. That selector builds the inference pool dynamically — no manual server inventory is required.

Many models on Hugging Face require an account and access token to download. The quickstart Helm chart provides a place to store such tokens as a Kubernetes Secret. This Quickstart used a public model (Qwen) so no secret is present. If you run gated models on your cluster, create the Kubernetes Secret but never print its value to the terminal.

Now send a real OpenAI-format completion request to the router (via the port-forward):

```bash theme={null}
curl -s http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","prompt":"How are you today?"}' | jq
```

Example JSON response (formatted by `jq`):

```json theme={null}
{
  "id": "cmpl-4697d272-adc1-4c96-832d-cddcef707592",
  "object": "text_completion",
  "created": 1786714656,
  "model": "Qwen/Qwen2.5-0.5B-Instruct",
  "choices": [
    {
      "index": 0,
      "text": " I'm a machine learning model, so my responses are based on algorithms and data",
      "logprobs": null,
      "finish_reason": "length",
      "stop_reason": null,
      "token_ids": null,
      "prompt_logprobs": null,
      "prompt_token_ids": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "prompt_tokens": 5,
    "total_tokens": 21,
    "completion_tokens": 16,
    "prompt_tokens_details": null
  },
  "kv_transfer_params": null
}
```

What happened end-to-end:

1. Envoy (inside the router pod) accepted the HTTP request at `localhost:8000`.
2. The EPP evaluated and scored the three available model server endpoints, picked one, and forwarded the request.
3. A model server produced the completion tokens and returned the response to the router, which returned it to your curl client.

Prometheus has been collecting metrics the whole time. With the Prometheus port-forward open on `localhost:9090`, query the router's view of the pool. The metric `llm_d_epp_ready_endpoints` reports how many ready endpoints the router knows about. A representative metric scrape immediately after the curl:

```text theme={null}
llm_d_epp_ready_endpoints

llm_d_epp_ready_endpoints{container="epp", endpoint="http-metrics", instance="10.244.223.194:9090", job="quickstart-epp", name="quickstart", namespace="llm-d", pod="quickstart-epp-5f6cbf8c4-tcl7q", service="quickstart-epp"} 3
```

* The router agrees with `kubectl get pods` — there are three ready model servers behind the frontend.
* If you run a Prometheus query for the request counter (e.g., the request-handling counter for the EPP) after the curl, you should see that counter increment by one.

This tour checks the full stack: Helm release, pods, standalone router (no Gateway), inference pool, the front door, and a real completion with a verifiable Prometheus trail.

Next: we'll slow a single request down to demonstrate pause, streaming, and scratchpad behavior.

References and further reading:

* [Kubernetes Documentation](https://kubernetes.io/docs/)
* [Prometheus](https://prometheus.io/)
* [Helm](https://helm.sh/)

> **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/cb6d2710-3350-448e-bc67-d78835a558a9/lesson/5affa85c-407a-4144-980c-fad61efb4ebd)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.