> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Calling the Model

> Shows how to port-forward to a KServe deployed model and call its OpenAI-compatible chat completions API using curl to receive JSON responses

Earlier we deployed the Qwen model and watched its status change from not-ready to ready. That deployment provides a live inference endpoint inside the Kubernetes cluster. In this lesson you'll open a connection to that endpoint using `kubectl port-forward`, send an HTTP request with `curl`, and inspect the JSON response. By the end you'll understand how to call a KServe inference endpoint that exposes an OpenAI-compatible API.

![The image shows the Qwen logo at the top. Below it are two status cards—on the left a red "False" (starting up) card with an X and on the right a green "True" (live endpoint) card with a check, connected by an arrow.](https://mintcdn.com/kodekloud-c4ac6d9a/2m51rHdQ_p1kNvO7/images/KServe-Fundamentals-Serving-ML-Models-on-Kubernetes/Serving-a-Generative-Model/Calling-the-Model/qwen-status-false-to-true.jpg?fit=max\&auto=format\&n=2m51rHdQ_p1kNvO7\&q=85\&s=05d4282c6c455dc75ccba26d6273f0e3)

When running inside a cluster, production traffic is typically routed through an ingress controller or API gateway to reach services. For local testing and learning we can create a direct tunnel from a local port to the Kubernetes Service using `kubectl port-forward`. Traffic you send to `localhost` is forwarded to the service inside the cluster for the duration of the command.

KServe creates an InferenceService plus a corresponding Kubernetes Service for the predictor. In this example the predictor service is named `qwen-model-predictor` and it lives in the `kserve-inference` namespace.

![A diagram titled "Opening the Tunnel" showing a Kubernetes cluster and a namespace called kserve-inference. Inside are two components: an InferenceService named qwen-model and a Kubernetes Service named qwen-model-predictor.](https://mintcdn.com/kodekloud-c4ac6d9a/2m51rHdQ_p1kNvO7/images/KServe-Fundamentals-Serving-ML-Models-on-Kubernetes/Serving-a-Generative-Model/Calling-the-Model/kserve-inference-qwen-model-predictor-diagram.jpg?fit=max\&auto=format\&n=2m51rHdQ_p1kNvO7\&q=85\&s=4a88cb32b21a13f8fc3003a40945b8dc)

Open a terminal and run this command (keep it running). It forwards local port `8080` to port `80` on the `qwen-model-predictor` service in the `kserve-inference` namespace:

```bash theme={null}
kubectl port-forward svc/qwen-model-predictor -n kserve-inference 8080:80
```

You should see output similar to:

```bash theme={null}
$ kubectl port-forward svc/qwen-model-predictor -n kserve-inference 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
```

Keep this terminal open while you make requests. The port-forwarding tunnel stops when you close the command.

Open a second terminal for sending requests to the local tunnel.

KServe's Hugging Face runtime exposes an OpenAI-compatible REST API. The chat completions endpoint is:

```text theme={null}
POST /openai/v1/chat/completions
```

A minimal JSON request body requires three fields: the model name, a list of `messages`, and a `max_tokens` limit to bound the response length. Example request (use this from the second terminal):

```bash theme={null}
curl -s http://localhost:8080/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-model",
    "messages": [
      {"role": "user", "content": "What is KServe in one sentence?"}
    ],
    "max_tokens": 100
  }'
```

Quick notes on the curl options and headers:

* `-s` silences curl’s progress output so only the response is printed.
* `-H "Content-Type: application/json"` ensures the server interprets the body as JSON.
* The JSON body includes the required `model`, `messages`, and `max_tokens` fields.

Request body fields (summary)

| Field | Purpose | Example |
| - | - | - |
| `model` | The model identifier deployed by KServe | `qwen-model` |
| `messages` | Array of message objects forming the chat history | `[{ "role": "user", "content": "Hello" }]` |
| `max_tokens` | Maximum number of tokens to generate for the completion | `100` |

The first request may take a few extra seconds as the model begins producing tokens. The server returns a JSON response that typically contains:

* `choices[0].message.content` — the model's textual reply you want to display or parse.
* `usage` — token accounting for prompt/completion/total (useful for cost monitoring).

For improved readability, pipe the response through `jq` (if installed):

```bash theme={null}
curl -s http://localhost:8080/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-model",
    "messages": [
      {"role": "user", "content": "What is KServe in one sentence?"}
    ],
    "max_tokens": 100
  }' | jq
```

Now you’ve sent a real request to a language model running inside Kubernetes and received a real JSON response.

![A simple diagram showing a client (laptop) sending a green "Request" arrow to a Kubernetes cluster hosting a "Live Language Model," which returns a pink "Response" arrow back to the client.](https://mintcdn.com/kodekloud-c4ac6d9a/2m51rHdQ_p1kNvO7/images/KServe-Fundamentals-Serving-ML-Models-on-Kubernetes/Serving-a-Generative-Model/Calling-the-Model/client-kubernetes-llm-request-response.jpg?fit=max\&auto=format\&n=2m51rHdQ_p1kNvO7\&q=85\&s=92b5be9d0b8c2b1d83f6b819ac93dee3)

That completes the full loop: deploy the model, expose it, and call it. Next, we'll inspect how KServe routes incoming requests inside the cluster and identify which container handles the inference traffic.

Links and references

* KServe docs: [https://kserve.github.io/website/](https://kserve.github.io/website/)
* kubectl port-forward: [https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/)
* Ingress (Kubernetes): [https://kubernetes.io/docs/concepts/services-networking/ingress/](https://kubernetes.io/docs/concepts/services-networking/ingress/)
* curl manual: [https://curl.se/docs/manpage.html](https://curl.se/docs/manpage.html)
* jq: [https://stedolan.github.io/jq/](https://stedolan.github.io/jq/)

> **[Watch Video](https://learn.kodekloud.com/user/courses/kserve-fundamentals-serving-ml-models-on-kubernetes/module/a270a816-b8e6-47f2-8ef9-7cf5b97f8890/lesson/b21c6662-bbf2-43f7-af4a-21a24e902cdf)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.