> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Demo Serving a Text Classifier
> Guide to deploying and testing a DistilBERT sentiment classifier on KServe with a manifest, port forwarding, OIP inference requests, and interpreting class probability responses.
Earlier we reviewed model distillation, inspected the manifest for our sentiment-classifier, and compared its structure to the Qwen deployment we examined previously. The overall layout and resources are nearly identical; the primary differences come from the runtime type and the inference protocol used by a classifier versus a conversational model.
This demo uses a manifest named `sentiment-classifier.yaml`. It creates an `InferenceService` called `sentiment-classifier` in the `kserve-inference` namespace and selects the Hugging Face runtime to load a DistilBERT-based model. The manifest includes a startup arg (for example `--return_all_scores`) so the runtime returns probability scores for each class (both negative and positive) instead of only the single winning class — that extra detail is visible in the response.
Apply the manifest to the cluster:
```bash theme={null}
less sentiment-classifier.yaml
kubectl apply -f sentiment-classifier.yaml
# Expected output:
# inferenceservice.serving.kserve.io/sentiment-classifier created
```
KServe will detect the new resource, select the Hugging Face runtime, and begin provisioning the pod (the initializer downloads the DistilBERT weights). Watch the InferenceService until it becomes ready:
```bash theme={null}
kubectl get inferenceservice sentiment-classifier -n kserve-inference -w
```
When the resource is initially created you will usually see `READY False` while the model downloads and the pod initializes. DistilBERT is relatively small (\~250 MB), so readiness often follows quickly, though actual timing depends on network and model registry availability. When `READY True` appears, the endpoint is live.
You may also see a route URL such as:
[http://sentiment-classifier-kserve-inference.example.com](http://sentiment-classifier-kserve-inference.example.com)
To access the service from your local machine, open a port-forward to the predictor service (run it in the background so you can keep using the same terminal for curl requests):
```bash theme={null}
kubectl port-forward svc/sentiment-classifier-predictor -n kserve-inference 8080:80 &
# Example output:
# Forwarding from 127.0.0.1:8080 -> 8080
```
Classifiers use an Open Inference Protocol (OIP) style endpoint rather than the OpenAI-compatible chat/completion endpoints used by conversational models (e.g., Qwen). Conversational models accept a `messages` array on endpoints like `/openai/v1/chat/completions`. Classifiers receive text inputs as OIP tensors via `/v2/models//infer`.
Endpoint (local port-forward):
/v2/models/sentiment-classifier/infer
Example curl request (OIP v2 format). This sends a single sentence to classify:
```bash theme={null}
curl -s http://localhost:8080/v2/models/sentiment-classifier/infer \
-H "Content-Type: application/json" \
-d '{
"inputs": [
{
"name": "input-0",
"shape": [1],
"datatype": "BYTES",
"data": ["I really enjoyed this course. The examples were clear and helpful."]
}
]
}' | jq .
```
Request field reference
| Field | Description | Example |
| ---------- | ---------------------------------------------- | ------------------ |
| `inputs` | Array of input tensors (OIP-style). | See request above. |
| `name` | Arbitrary identifier for the input tensor. | `"input-0"` |
| `shape` | Tensor shape; `[1]` for a single sentence. | `[1]` |
| `datatype` | Data type of the tensor. Use `BYTES` for text. | `BYTES` |
| `data` | Array of text strings to classify. | `["I loved it!"]` |
Sample response (formatted):
```json theme={null}
{
"model_name": "sentiment-classifier",
"model_version": null,
"id": "31f71876-e616-4aeb-a9d4-4c1f27f8efe1",
"parameters": null,
"outputs": [
{
"name": "output-0",
"shape": [1],
"datatype": "BYTES",
"parameters": null,
"data": [
"{0: 0.0001, 1: 0.9999}"
]
}
]
}
```
How to interpret the response:
* `outputs[0].data[0]` contains a string encoding of class probabilities mapping class index → probability.
* In this example, key `0` = negative sentiment and key `1` = positive sentiment.
* The example indicates \~0.0001 probability for negative and \~0.9999 for positive — \~99.99% confidence that the sentence is positive.
Try different inputs to observe how confidence varies:
* Strongly negative: "This course is a waste of time. Nothing worked the way it was supposed to." → high probability for `0` (negative).
* Ambiguous: "It was fine, I guess." → probabilities closer together (e.g., \~60% vs \~40%), showing model uncertainty that would be hidden by a bare class label.
Quick lab steps (try this yourself)
| Step | Command |
| ------------------------- | --------------------------------------------------------------------------------------- |
| 1. Apply manifest | `kubectl apply -f sentiment-classifier.yaml` |
| 2. Wait for readiness | `kubectl get inferenceservice sentiment-classifier -n kserve-inference -w` |
| 3. Port-forward predictor | `kubectl port-forward svc/sentiment-classifier-predictor -n kserve-inference 8080:80 &` |
| 4. Send inference request | Use the curl example above to POST to `/v2/models/sentiment-classifier/infer` |
| 5. Inspect result | Check `outputs[0].data[0]` — `0` = negative, `1` = positive |
If you run port-forward in the background, remember to kill the job when finished (e.g., `kill %1` or use `pkill -f "kubectl port-forward ..."`). Leaving processes running can keep ports occupied or leak credentials in some environments.
Observability tips
* Clear, unambiguous inputs tend to produce high-confidence scores (>99%).
* Ambiguous or neutral sentences produce probabilities that are closer together, which is exactly why returning scores (instead of just labels) gives better insight into model uncertainty.
* Use multiple example sentences to validate expected behavior across edge cases.
References
* [KServe Documentation](https://kserve.github.io/website/)
* [Hugging Face Models & Runtimes](https://huggingface.co/)
* [Kubernetes Basics](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/)
Good luck experimenting — vary the inputs and observe how the probability distribution reflects confidence and ambiguity.