Demo Serving a Text Classifier

Earlier we reviewed model distillation, inspected the manifest for our sentiment-classifier, and compared its structure to the Qwen deployment we examined previously. The overall layout and resources are nearly identical; the primary differences come from the runtime type and the inference protocol used by a classifier versus a conversational model.

This demo uses a manifest named sentiment-classifier.yaml. It creates an InferenceService called sentiment-classifier in the kserve-inference namespace and selects the Hugging Face runtime to load a DistilBERT-based model. The manifest includes a startup arg (for example --return_all_scores) so the runtime returns probability scores for each class (both negative and positive) instead of only the single winning class — that extra detail is visible in the response. Apply the manifest to the cluster:

less sentiment-classifier.yaml
kubectl apply -f sentiment-classifier.yaml
# Expected output:
# inferenceservice.serving.kserve.io/sentiment-classifier created

KServe will detect the new resource, select the Hugging Face runtime, and begin provisioning the pod (the initializer downloads the DistilBERT weights). Watch the InferenceService until it becomes ready:

kubectl get inferenceservice sentiment-classifier -n kserve-inference -w

When the resource is initially created you will usually see READY False while the model downloads and the pod initializes. DistilBERT is relatively small (~250 MB), so readiness often follows quickly, though actual timing depends on network and model registry availability. When READY True appears, the endpoint is live. You may also see a route URL such as: http://sentiment-classifier-kserve-inference.example.com To access the service from your local machine, open a port-forward to the predictor service (run it in the background so you can keep using the same terminal for curl requests):

kubectl port-forward svc/sentiment-classifier-predictor -n kserve-inference 8080:80 &
# Example output:
# Forwarding from 127.0.0.1:8080 -> 8080

Classifiers use an Open Inference Protocol (OIP) style endpoint rather than the OpenAI-compatible chat/completion endpoints used by conversational models (e.g., Qwen). Conversational models accept a messages array on endpoints like /openai/v1/chat/completions. Classifiers receive text inputs as OIP tensors via /v2/models/<model-name>/infer.

Endpoint (local port-forward): /v2/models/sentiment-classifier/infer Example curl request (OIP v2 format). This sends a single sentence to classify:

curl -s http://localhost:8080/v2/models/sentiment-classifier/infer \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": [
      {
        "name": "input-0",
        "shape": [1],
        "datatype": "BYTES",
        "data": ["I really enjoyed this course. The examples were clear and helpful."]
      }
    ]
  }' | jq .

Request field reference

Field	Description	Example
`inputs`	Array of input tensors (OIP-style).	See request above.
`name`	Arbitrary identifier for the input tensor.	`"input-0"`
`shape`	Tensor shape; `[1]` for a single sentence.	`[1]`
`datatype`	Data type of the tensor. Use `BYTES` for text.	`BYTES`
`data`	Array of text strings to classify.	`["I loved it!"]`

Sample response (formatted):

{
  "model_name": "sentiment-classifier",
  "model_version": null,
  "id": "31f71876-e616-4aeb-a9d4-4c1f27f8efe1",
  "parameters": null,
  "outputs": [
    {
      "name": "output-0",
      "shape": [1],
      "datatype": "BYTES",
      "parameters": null,
      "data": [
        "{0: 0.0001, 1: 0.9999}"
      ]
    }
  ]
}

How to interpret the response:

outputs[0].data[0] contains a string encoding of class probabilities mapping class index → probability.
In this example, key 0 = negative sentiment and key 1 = positive sentiment.
The example indicates ~0.0001 probability for negative and ~0.9999 for positive — ~99.99% confidence that the sentence is positive.

Try different inputs to observe how confidence varies:

Strongly negative: “This course is a waste of time. Nothing worked the way it was supposed to.” → high probability for 0 (negative).
Ambiguous: “It was fine, I guess.” → probabilities closer together (e.g., ~60% vs ~40%), showing model uncertainty that would be hidden by a bare class label.

Quick lab steps (try this yourself)

Step	Command
1. Apply manifest	`kubectl apply -f sentiment-classifier.yaml`
2. Wait for readiness	`kubectl get inferenceservice sentiment-classifier -n kserve-inference -w`
3. Port-forward predictor	`kubectl port-forward svc/sentiment-classifier-predictor -n kserve-inference 8080:80 &`
4. Send inference request	Use the curl example above to POST to `/v2/models/sentiment-classifier/infer`
5. Inspect result	Check `outputs[0].data[0]` — `0` = negative, `1` = positive

If you run port-forward in the background, remember to kill the job when finished (e.g., kill %1 or use pkill -f "kubectl port-forward ..."). Leaving processes running can keep ports occupied or leak credentials in some environments.

Observability tips

Clear, unambiguous inputs tend to produce high-confidence scores (>99%).
Ambiguous or neutral sentences produce probabilities that are closer together, which is exactly why returning scores (instead of just labels) gives better insight into model uncertainty.
Use multiple example sentences to validate expected behavior across edge cases.

References

Good luck experimenting — vary the inputs and observe how the probability distribution reflects confidence and ambiguity.

Foundations

Serving a Generative Model

Serving a Predictive Model

Troubleshooting Basics

Demo Serving a Text Classifier

Watch Video

Practice Lab