Serving a Quantized Qwen Model

The InferenceService manifest—the YAML that tells KServe what model to deploy and how to run it—defines both deployment and runtime configuration. This lesson covers a concrete example: serving a small, quantized Qwen model from Hugging Face. By the end you’ll understand:

what “quantized” means,
why it’s a good fit for a learning environment, and
the end-to-end sequence of events that occur after you apply the manifest to your cluster.

When a language model is trained, its internal parameters (weights) are stored as high-precision numbers (for example, float32). That precision helps during training but consumes significant memory. Quantization reduces that precision after training finishes: weights are converted to lower-precision representations (for example, int8 or other compact formats). The trade-off is much smaller model size and lower memory usage at the cost of a modest drop in accuracy—exactly the trade-off we want for CPU-based learning and demo environments.

A presentation slide titled "Objective" with a vertical, numbered list of items. The points mention understanding the Qwen model from Hugging Face, what a quantized model means, benefits of quantized models in a learning environment, and a step‑by‑step flow after applying the manifest to the cluster.

Quantization is ideal here because the model can run on commodity CPU nodes without GPUs or specialized runtimes and still provide real, testable language outputs. For this demo we use Qwen2.5-0.5B-Instruct — a 0.5B-parameter variant that has been quantized to run comfortably on CPU. It’s compact by contemporary standards but excellent for demonstrations and learning.

A slide titled "Quantization – Introduction" with the text "Quantization is the process of reducing that precision after training is done." A small diagram shows a numeric value being rounded (0.1234567891 → 0.123) from a model to illustrate reduced precision and less memory usage.

Below is the InferenceService manifest we will apply. It instructs KServe to use the Hugging Face model format and points the storage initializer to the Hugging Face repository using the hf:// URI prefix.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: qwen-model
  namespace: kserve-inference
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      storageUri: "hf://Qwen/Qwen2.5-0.5B-Instruct"
      args:
        - --backend=huggingface
      resources:
        requests:
          cpu: "1"
          memory: "2Gi"
        limits:
          cpu: "2"
          memory: "6Gi"

Why hf://? The hf:// prefix tells KServe’s storage initializer to fetch the model directly from Hugging Face (no external object storage needed). The storage initializer runs as an init container inside the pod and downloads model artifacts onto the pod’s local filesystem so the serving container can load them. Resource requests and limits are tuned for this quantized model to run on CPU without over-allocating cluster capacity:

Requests: 1 CPU, 2Gi memory
Limits: 2 CPU, 6Gi memory

Resource	Request	Limit
CPU	`1`	`2`
Memory	`2Gi`	`6Gi`

Quantized models trade a small amount of accuracy for much lower memory usage and faster CPU inference. For development, demos, and learning environments this trade-off is often acceptable; for production workloads evaluate accuracy, latency, and failure modes carefully before deploying.

First steps — save and apply the manifest:

Save the YAML above as qwen-model.yaml.
Create a dedicated namespace and apply the manifest:

kubectl create namespace kserve-inference

kubectl apply -f qwen-model.yaml

Expected output immediately after creating the namespace and applying the manifest:

$ kubectl create ns kserve-inference
namespace/kserve-inference created

$ kubectl apply -f qwen-model.yaml
inferenceservice.serving.kserve.io/qwen-model created

That single apply triggers a series of actions inside the cluster:

The KServe controller detects the new InferenceService.
It chooses the appropriate runtime (here, a Hugging Face runtime).
The controller creates a pod that includes an init (storage initializer) container plus the main serving container.

At startup, the storage initializer (init container) downloads the model files from Hugging Face into the pod’s filesystem. Once the download completes, the init container exits and the main serving container starts and loads the weights into memory.

A diagram titled "Watching the Startup" showing a Hugging Face model repository at the top and two boxes: a Storage Initializer with Model Weights on the left and a Serving Container with Memory on the right. An arrow labeled "Transfer weights for loading" indicates moving the weights from storage to the serving container.

How to monitor progress:

Check the InferenceService status:

kubectl get inferenceservice qwen-model -n kserve-inference

Shortly after applying you’ll typically see the resource listed but not yet ready:

$ kubectl get inferenceservice qwen-model -n kserve-inference
NAME       URL   READY   PREV   LATEST   AGE
qwen-model        False                          30s

Watch pods to observe init container activity and when the serving container starts:

kubectl get pods -n kserve-inference -w

You will see two phases:

Init (storage initializer) container runs and downloads model weights.
Main serving container starts after the init container completes and loads the model into memory.

When download and model-loading finish, the InferenceService status will flip to Ready and the service will publish a cluster-local URL:

$ kubectl get inferenceservice qwen-model -n kserve-inference
NAME       URL                                           READY   PREV   LATEST   AGE
qwen-model http://qwen-model.kserve.svc.cluster.local    True                       4m

Once Ready is True the model is live inside the cluster and will accept requests at that URL. For local testing you can use kubectl port-forward to reach the service; in production you’ll typically expose it via an ingress, API gateway, or service mesh.

A diagram titled "What You Now Have" showing a green "Ready / True" live inference endpoint. Below it is a dashed Kubernetes cluster box containing a Qwen model.

KServe provides a consistent inference interface across runtimes. After the InferenceService becomes Ready you can send inference requests using the runtime’s expected POST format (Hugging Face runtime formats inputs similarly to the HF Inference API). For local experiments, curl or a small Python client are usually sufficient. Summary — startup flow

Create namespace and apply the InferenceService manifest.
KServe controller creates a pod with an init (storage initializer) and a serving container.
Init container downloads model artifacts from Hugging Face to the pod.
Serving container loads the model into memory and begins accepting requests.
InferenceService status becomes Ready and the cluster service URL becomes available.

This example is tuned for demos and learning. Quantized models reduce accuracy slightly. For production deployments validate model quality, load-test latency, and add readiness/liveness, autoscaling, and security controls before exposing the endpoint externally.

Links and references

This quantized Qwen model provides a compact, CPU-friendly inference endpoint—ideal for demos and learning while producing usable language model responses.

Foundations

Serving a Generative Model

Serving a Predictive Model

Troubleshooting Basics

Serving a Quantized Qwen Model

Watch Video