Demo Deploying and Calling the Qwen Model

In this lesson you’ll deploy the Qwen model to a Kubernetes cluster using KServe and call it via an OpenAI-compatible endpoint. The walkthrough covers creating the namespace, applying the InferenceService manifest, waiting for the model to become READY, port-forwarding the predictor service to your workstation, and sending a chat completion request.

Quick flow for the lab

Create namespace kserve-inference.
Inspect the InferenceService manifest (qwen-model.yaml) for the Qwen model.
Apply the manifest to create the InferenceService.
Wait until the InferenceService reports READY=true.
Port-forward the predictor service to your local machine.
Send an OpenAI-compatible chat completion request and inspect the response.

Prerequisite — verify your cluster

kubectl get nodes

Example output:

NAME           STATUS   ROLES           AGE   VERSION
minikube       Ready    control-plane   96m   v1.35.1

Create the namespace

kubectl create namespace kserve-inference

Example output:

namespace/kserve-inference created

Inspect the InferenceService manifest The qwen-model.yaml creates an InferenceService named qwen-model in the kserve-inference namespace and configures a Hugging Face–format model. The manifest below shows the predictor configuration, storage URI, and resource requests/limits.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: qwen-model
  namespace: kserve-inference
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      storageUri: "hf://Qwen/Qwen2.5-0.5B-Instruct"
      args:
        - --backend=huggingface
      resources:
        requests:
          cpu: "1"
          memory: "2Gi"
        limits:
          cpu: "2"
          memory: "6Gi"

Apply the manifest

kubectl apply -f qwen-model.yaml

Example output:

inferenceservice.serving.kserve.io/qwen-model created

Watch the InferenceService status Wait until KServe finishes creating the InferenceService and the model is ready. Use the -w (watch) flag to observe status changes in real time:

kubectl get inferenceservice qwen-model -n kserve-inference -w

Trimmed example progression:

NAME        URL                                                         READY   AGE
qwen-model                                                                  False  18s
qwen-model  http://qwen-model-kserve-inference.example.com            True   104s

Wait until READY reports True before attempting to connect. Trying earlier can yield connection errors or model-startup failures.

Confirm the predictor service Once READY=true, confirm the predictor service exists — this is the service you’ll port-forward to:

kubectl get service -n kserve-inference

Example output:

NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
qwen-model-predictor    ClusterIP   10.109.99.21   <none>        80/TCP     2m2s

Port-forward the predictor service Create a local tunnel from your workstation to the predictor service. Run this in the background so it stays open while you send requests:

kubectl port-forward svc/qwen-model-predictor -n kserve-inference 8080:80 &

Example shell output:

[1] 12956
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

Port 8080 on your workstation now forwards to port 80 on the predictor service inside the cluster. Send a chat completion request (OpenAI-compatible) Use the OpenAI-compatible Chat Completions endpoint exposed by KServe. This request follows the OpenAI Chat Completions schema and targets the local port-forward.

curl -s http://localhost:8080/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-model",
    "messages": [
      {"role": "user", "content": "What is KServe in one sentence?"}
    ],
    "max_tokens": 100
  }' | jq

Note: while the model is generating, the curl command may appear to hang briefly — this is expected while tokens are produced. Example (trimmed) JSON response:

{
  "id": "a92024e8-3b20-4d6c-98c2-ffe3f3034245",
  "object": "chat.completion",
  "created": 1779894025,
  "model": "qwen-model",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "KServe is a cloud-native, scalable, and secure data lake and data warehouse platform designed to help organizations manage, analyze, and visualize their data efficiently.",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "tool_calls": [],
        "reasoning": null,
        "reasoning_content": null
      },
      "logprobs": null,
      "finish_reason": "length",
      "stop_reason": null,
      "token_ids": null
    }
  ],
  "usage": {
    "prompt_tokens": 37,
    "total_tokens": 69,
    "completion_tokens": 32
  }
}

Response highlights

The assistant reply is located at choices[0].message.content.
The usage block contains token counts (useful for tracking cost and quotas).

Quick commands reference

Action	Command
Create namespace	`kubectl create namespace kserve-inference`
Apply manifest	`kubectl apply -f qwen-model.yaml`
Watch InferenceService	`kubectl get inferenceservice qwen-model -n kserve-inference -w`
List services	`kubectl get service -n kserve-inference`
Port-forward predictor	`kubectl port-forward svc/qwen-model-predictor -n kserve-inference 8080:80 &`
Send chat completion	`curl -s http://localhost:8080/openai/v1/chat/completions -H "Content-Type: application/json" -d '<payload>' \| jq`

Lab checklist

Verify the Qwen InferenceService reports READY=true in the kserve-inference namespace.
Start a port-forward to the qwen-model-predictor service (local 8080 -> service 80).
Send the curl request to http://localhost:8080/openai/v1/chat/completions as shown above.
Inspect choices[0].message.content to read the model’s reply.

Links and references

KServe docs: https://kserve.github.io/
Hugging Face: https://huggingface.co
Kubernetes docs: https://kubernetes.io/docs/

Foundations

Serving a Generative Model

Serving a Predictive Model

Troubleshooting Basics

Demo Deploying and Calling the Qwen Model

Watch Video

Practice Lab