What Changes With Predictive Models

Welcome to this module on serving predictive models with KServe. In the previous lesson we deployed a generative language model (Qwen) and invoked it with chat-style prompts. In this lesson we’ll contrast that workflow with predictive (classification) models: what changes in the request/response shape, what stays the same in your deployment manifest, and why distilled models like DistilBERT are a great fit for production classification tasks. This guide covers:

The conceptual difference between generative and predictive models
Why request and response payloads differ
How KServe keeps the InferenceService manifest stable across model types
A brief introduction to model distillation and the DistilBERT sentiment classifier we’ll deploy

The image shows a presentation agenda with three points: differentiating predictive models from generative models, explaining model type impacts on request/response formats, and describing KServe InferenceService manifest stability.

Generative vs Predictive — core concept

A generative model composes new content token-by-token. Example: you ask “Tell me about Kubernetes” and the model generates an explanatory paragraph. Its output space is effectively open-ended. A predictive model, by contrast, maps inputs to a predefined set of labels. It answers questions like “Which bucket does this input belong to?” or “Which label best describes this input?” The output is a discrete set (e.g., ) or a probability distribution over those labels.

The image illustrates the core difference between generative and predictive models, showing input from a client going into a predictive model to determine which category it belongs to, resulting in output from a fixed set of answers.

Common predictive examples:

Spam filtering: label an email spam or not_spam.

The image illustrates a spam filtering process using a predictive model, showing email as input and classifying the output as either "Spam" or "Not spam."

Image classification: detect whether a photo contains a cat (cat / no_cat).
Sentiment analysis: classify text as positive or negative.

The classifier we’ll deploy in this lesson performs sentiment analysis: it reads a sentence and predicts positive or negative sentiment. This is a canonical predictive task and useful to illustrate the differences from generative workflows.

The image displays a flowchart of a predictive model, where a client input ("This course has been great!") is processed to produce either a positive or negative output. It illustrates how the model predicts outcomes based on given inputs.

Distillation — why DistilBERT?

Distillation trains a smaller “student” model to mimic a larger “teacher”. The result is a compact model that approximates the teacher’s behavior while using far less memory, CPU, and storage. Distilled models trade a small amount of accuracy for much lower inference cost — ideal for production classifiers.

The image explains distillation in machine learning, illustrating how a smaller model is trained to mimic a larger one, with visual representations of a large model, distillation process, and distilled model.

The model we’ll use is:

Name: distilbert-base-uncased-finetuned-sst-2-english
- DistilBERT: lighter, faster variant of BERT
- uncased: ignores case differences
- finetuned SST-2 English: trained on the Stanford Sentiment Treebank (SST-2) for sentiment classification

When you send a sentence to this model it returns a label (positive/negative) and a confidence score. We pull the model directly from Hugging Face via an hf URI — no external object storage is required. The Hugging Face model URI is: hf://distilbert-base-uncased-finetuned-sst-2-english

KServe InferenceService — what stays the same

One of KServe’s strengths is that the InferenceService manifest and predictor spec look the same whether you deploy a large generative model or a compact classifier. Only a few fields change: most notably the InferenceService metadata.name and the storageUri (model identifier). Resource requests/limits typically shrink for distilled classifiers. Example InferenceService manifest (Hugging Face model):

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sentiment-classifier
  namespace: kserve-inference
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      storageUri: "hf://distilbert-base-uncased-finetuned-sst-2-english"
      resources:
        requests:
          cpu: "500m"
          memory: "1Gi"
        limits:
          cpu: "1"
          memory: "2Gi"

Only a few manifest details differ compared to a generative model deployment:

metadata.name — a descriptive name for the InferenceService
storageUri — model identifier (e.g., the Hugging Face URI)
resources — classifiers typically require smaller requests/limits than large generative LLMs

This consistency means you can reuse the same deployment workflow and tooling across generative and predictive models.

Request / Response shape — what changes

The primary runtime difference is the inference payload. Generative models accept chat-style prompts (a list of messages, system and user roles) and produce open-ended text. Classifiers accept a single input (text or data tensor) and return a label or a probability distribution over labels. Common payload examples (these are typical shapes — exact format depends on your predictor and KServe runtime configuration):

Simple instances-style request:

{
  "instances": [
    "This course has been great!"
  ]
}

Typical classifier response (probabilities for labels in fixed order [negative, positive]):

{
  "predictions": [
    [0.02, 0.98]
  ]
}

Alternatively, some Hugging Face-style APIs return labeled results:

[
  {
    "label": "POSITIVE",
    "score": 0.9873
  }
]

The image compares "Qwen" and "Classifier" request formats, highlighting that Qwen uses chat-style prompts, accepts system and user messages, and generates text responses, whereas the Classifier takes plain text input and returns labels or categories without needing a conversation format.

Note: the exact JSON keys (instances vs inputs vs a Hugging Face-specific shape) can differ by predictor and the chosen protocol (V1/V2). KServe docs and your runtime configuration specify the exact expected format.

Key takeaway: predictive models select from a predefined set of labels (e.g., positive/negative). KServe uses the same InferenceService manifest and runtime for both generative and predictive models; only the inference payload shape changes.

Quick comparison: generative vs predictive

Aspect	Generative Model	Predictive (Classifier)
Output	Free-form text	Fixed set of labels or probabilities
Input	Chat-style messages, prompts	Single text or tensor input
Typical use	Content generation, completion	Classification, detection, ranking
Resource cost	Often very high (large LLMs)	Typically lower (distilled models)
KServe manifest	Same API & predictor spec	Same API & predictor spec

Summary

Predictive models map inputs to predefined labels rather than generating free-form content.
DistilBERT (fine-tuned on SST-2) is a compact, production-friendly sentiment classifier.
KServe keeps the InferenceService manifest and runtime consistent across different model types; only the model URI and resource sizing change.
The main difference is the runtime payload format: generative models accept prompts/messages, while classifiers expect a single input and return labels or probability distributions.

Next: we’ll deploy the sentiment classifier and demonstrate the exact request/response payloads for your KServe setup. References:

KServe documentation: https://kserve.github.io
Hugging Face models: https://huggingface.co
Stanford Sentiment Treebank (SST-2): https://nlp.stanford.edu/sentiment/

Foundations

Serving a Generative Model

Serving a Predictive Model

Troubleshooting Basics

What Changes With Predictive Models

Generative vs Predictive — core concept

Distillation — why DistilBERT?

KServe InferenceService — what stays the same

Request / Response shape — what changes

Quick comparison: generative vs predictive

Summary

Watch Video

​Generative vs Predictive — core concept

​Distillation — why DistilBERT?

​KServe InferenceService — what stays the same

​Request / Response shape — what changes

​Quick comparison: generative vs predictive

​Summary

Watch Video

Generative vs Predictive — core concept

Distillation — why DistilBERT?

KServe InferenceService — what stays the same

Request / Response shape — what changes

Quick comparison: generative vs predictive

Summary