> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo Deploying and Calling the Qwen Model

> Guide to deploying the Qwen model on Kubernetes with KServe and invoking it through an OpenAI compatible endpoint, including InferenceService setup, port forwarding, and chat request examples.

In this lesson you'll deploy the Qwen model to a Kubernetes cluster using KServe and call it via an OpenAI-compatible endpoint. The walkthrough covers creating the namespace, applying the InferenceService manifest, waiting for the model to become READY, port-forwarding the predictor service to your workstation, and sending a chat completion request.

![A presentation slide titled "Demo: Deploying and Calling the Qwen Model." The right side has a dark curved panel with a line-art rocket launching from a laptop.](https://mintcdn.com/kodekloud-c4ac6d9a/2m51rHdQ_p1kNvO7/images/KServe-Fundamentals-Serving-ML-Models-on-Kubernetes/Serving-a-Generative-Model/Demo-Deploying-and-Calling-the-Qwen-Model/qwen-deploy-demo-rocket-laptop.jpg?fit=max\&auto=format\&n=2m51rHdQ_p1kNvO7\&q=85\&s=8295cc5841dad3bb031a4f30717e74cc)

Quick flow for the lab

1. Create namespace `kserve-inference`.
2. Inspect the InferenceService manifest (`qwen-model.yaml`) for the Qwen model.
3. Apply the manifest to create the InferenceService.
4. Wait until the InferenceService reports `READY=true`.
5. Port-forward the predictor service to your local machine.
6. Send an OpenAI-compatible chat completion request and inspect the response.

Prerequisite — verify your cluster

```bash theme={null}
kubectl get nodes
```

Example output:

```text theme={null}
NAME           STATUS   ROLES           AGE   VERSION
minikube       Ready    control-plane   96m   v1.35.1
```

Create the namespace

```bash theme={null}
kubectl create namespace kserve-inference
```

Example output:

```text theme={null}
namespace/kserve-inference created
```

Inspect the InferenceService manifest
The `qwen-model.yaml` creates an InferenceService named `qwen-model` in the `kserve-inference` namespace and configures a Hugging Face–format model. The manifest below shows the predictor configuration, storage URI, and resource requests/limits.

```yaml theme={null}
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: qwen-model
  namespace: kserve-inference
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      storageUri: "hf://Qwen/Qwen2.5-0.5B-Instruct"
      args:
        - --backend=huggingface
      resources:
        requests:
          cpu: "1"
          memory: "2Gi"
        limits:
          cpu: "2"
          memory: "6Gi"
```

Apply the manifest

```bash theme={null}
kubectl apply -f qwen-model.yaml
```

Example output:

```text theme={null}
inferenceservice.serving.kserve.io/qwen-model created
```

Watch the InferenceService status
Wait until KServe finishes creating the InferenceService and the model is ready. Use the `-w` (watch) flag to observe status changes in real time:

```bash theme={null}
kubectl get inferenceservice qwen-model -n kserve-inference -w
```

Trimmed example progression:

```text theme={null}
NAME        URL                                                         READY   AGE
qwen-model                                                                  False  18s
qwen-model  http://qwen-model-kserve-inference.example.com            True   104s
```

Wait until `READY` reports `True` before attempting to connect. Trying earlier can yield connection errors or model-startup failures.

Confirm the predictor service
Once `READY=true`, confirm the predictor service exists — this is the service you'll port-forward to:

```bash theme={null}
kubectl get service -n kserve-inference
```

Example output:

```text theme={null}
NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
qwen-model-predictor    ClusterIP   10.109.99.21   <none>        80/TCP     2m2s
```

Port-forward the predictor service
Create a local tunnel from your workstation to the predictor service. Run this in the background so it stays open while you send requests:

```bash theme={null}
kubectl port-forward svc/qwen-model-predictor -n kserve-inference 8080:80 &
```

Example shell output:

```text theme={null}
[1] 12956
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
```

Port `8080` on your workstation now forwards to port `80` on the predictor service inside the cluster.

Send a chat completion request (OpenAI-compatible)
Use the OpenAI-compatible Chat Completions endpoint exposed by KServe. This request follows the OpenAI Chat Completions schema and targets the local port-forward.

```bash theme={null}
curl -s http://localhost:8080/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-model",
    "messages": [
      {"role": "user", "content": "What is KServe in one sentence?"}
    ],
    "max_tokens": 100
  }' | jq
```

Note: while the model is generating, the curl command may appear to hang briefly — this is expected while tokens are produced.

Example (trimmed) JSON response:

```json theme={null}
{
  "id": "a92024e8-3b20-4d6c-98c2-ffe3f3034245",
  "object": "chat.completion",
  "created": 1779894025,
  "model": "qwen-model",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "KServe is a cloud-native, scalable, and secure data lake and data warehouse platform designed to help organizations manage, analyze, and visualize their data efficiently.",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "tool_calls": [],
        "reasoning": null,
        "reasoning_content": null
      },
      "logprobs": null,
      "finish_reason": "length",
      "stop_reason": null,
      "token_ids": null
    }
  ],
  "usage": {
    "prompt_tokens": 37,
    "total_tokens": 69,
    "completion_tokens": 32
  }
}
```

Response highlights

* The assistant reply is located at `choices[0].message.content`.
* The `usage` block contains token counts (useful for tracking cost and quotas).

Quick commands reference

| Action | Command |
| - | - |
| Create namespace | `kubectl create namespace kserve-inference` |
| Apply manifest | `kubectl apply -f qwen-model.yaml` |
| Watch InferenceService | `kubectl get inferenceservice qwen-model -n kserve-inference -w` |
| List services | `kubectl get service -n kserve-inference` |
| Port-forward predictor | `kubectl port-forward svc/qwen-model-predictor -n kserve-inference 8080:80 &` |
| Send chat completion | `curl -s http://localhost:8080/openai/v1/chat/completions -H "Content-Type: application/json" -d '<payload>' \| jq` |

Lab checklist

* Verify the Qwen InferenceService reports `READY=true` in the `kserve-inference` namespace.
* Start a port-forward to the `qwen-model-predictor` service (local `8080` -> service `80`).
* Send the curl request to `http://localhost:8080/openai/v1/chat/completions` as shown above.
* Inspect `choices[0].message.content` to read the model's reply.

Links and references

* KServe docs: [https://kserve.github.io/](https://kserve.github.io/)
* Hugging Face: [https://huggingface.co](https://huggingface.co)
* Kubernetes docs: [https://kubernetes.io/docs/](https://kubernetes.io/docs/)

> **[Watch Video](https://learn.kodekloud.com/user/courses/kserve-fundamentals-serving-ml-models-on-kubernetes/module/a270a816-b8e6-47f2-8ef9-7cf5b97f8890/lesson/298807a3-6418-4daa-b274-b7591339abfc)**

> **[Practice Lab](https://learn.kodekloud.com/user/courses/kserve-fundamentals-serving-ml-models-on-kubernetes/module/a270a816-b8e6-47f2-8ef9-7cf5b97f8890/lesson/463037d1-fb3b-4af0-80a8-27014aa0508c)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.