Introduction to KServe and practical model serving on Kubernetes, covering deploying Hugging Face models, inference services, operational challenges, and differences between predictive and generative workloads.
Welcome to KServe Fundamentals.This course focuses on a practical — and often underappreciated — part of the machine learning lifecycle: taking a model out of a notebook and into a system that actually uses it.
Hi, I’m Chris Short — a longtime DevOps practitioner, Kubernetes contributor, and author of the DevOps’ish newsletter. I’m your instructor for this course. My goal is to help you not only get started with KServe, but also build the confidence to operate it in your environments.You don’t need to be a Kubernetes expert to participate, but you should be familiar with core Kubernetes concepts and comfortable using kubectl to manage a cluster.
You should also be comfortable using Helm to install KServe itself.At its simplest, a model is a function: give it input (an image, a row of data, a sentence) and it returns a prediction — a label, a number, a probability, or a generated response.
A model artifact — whether you trained it yourself or downloaded it — is typically a bundle of files on disk: weights, configuration, tokenizer files, and so on. Those files alone don’t accept requests or return predictions; they need an execution environment that loads them and exposes an API.Hugging Face has become a common home for model artifacts — often described as the “GitHub for AI” — with tens of thousands of models across many tasks and frameworks. Throughout this course we’ll rely on real, open models instead of a bespoke model trained for months. Specifically, we’ll use Qwen (an open-capable model from Alibaba on Hugging Face) and later a DistilBERT-based sentiment classifier.
What does a model server do?
Loads the model artifact into memory for fast inference.
Exposes an API endpoint (typically HTTP and/or gRPC) for client requests.
Manages the request lifecycle: receive input, preprocess, run inference, post-process, and return results.
At scale, handles batching, concurrency, health checks, autoscaling, and graceful restarts.
Model serving is the process of deploying a trained ML model so it can receive requests, run inference, and return predictions in real time or near-real time. Conceptually, it’s like any other web service: rather than returning HTML or fetching JSON from a database, the service runs a model on provided inputs and returns predictions.
How do you make a model artifact available so other systems — a web app, mobile client, pipeline, or API consumer — can call it? In this course we’ll:
Install KServe on Kubernetes in standard mode using Helm and cert-manager.
Use open models from Hugging Face (Qwen and DistilBERT).
Create InferenceService manifests in YAML to declare the models we want to serve.
Use curl and jq to interact with the served models during demos and testing.
Recommended prerequisites: familiarity with core Kubernetes concepts, basic kubectl usage, and Helm. You should also be comfortable using the command line for testing (curl, jq). These skills will help you follow the hands-on demos more effectively.
If you’ve wrapped a model in a simple Flask app and called it a day, you probably discovered the gap between “it works on my laptop” and “it works reliably under real traffic.” Common real-world serving challenges include:
Resource intensity: large models often need GPUs or specialized inference hardware to meet latency targets.
Unpredictable scale: traffic spikes require rapid scaling, and you need policies to reduce costs during low traffic.
Many models: organizations often serve dozens or hundreds of models, which requires repeatable deployment, configuration, and monitoring.
Concurrency and memory management: handling concurrent requests and multiple models requires careful orchestration.
Open model availability (for example, pulling models from Hugging Face) only solves artifact distribution — running models reliably at scale, managing concurrency, and controlling compute costs are still difficult. KServe is designed to address these operational challenges.
One important distinction to draw early is between two broad classes of models: predictive and generative.Predictive models include classifiers (image labeling), regressors (price prediction), recommenders (scoring items), or sentiment analysis. These models typically accept a request and return a single prediction. They tend to be faster, less resource-intensive, and stateless.Generative models include large language models (LLMs) and image-generation models. These are often much larger and compute-heavy, may stream outputs incrementally, and require advanced batching and memory strategies.To help compare, here’s a concise breakdown:
Characteristic
Predictive Models
Generative Models
Typical output
Single label/score
Streams or token sequences
Resource needs
Lower (often CPU)
Higher (GPU / large memory)
Latency pattern
Short, consistent
Variable, possibly streaming
Common tasks
Classification, regression, recommendation
LLMs, text/image generation
This course covers both flavors. Early on we’ll use a quantized, CPU-friendly Qwen model from Hugging Face for generative inference. Later we’ll deploy a DistilBERT-based text classifier to demonstrate how the same KServe primitives apply to predictive workloads. You’ll see how operational considerations change when moving from a language model to a lightweight classifier.
Summary: model serving is the bridge between a trained model artifact on disk and a live system that uses that model. Serving makes the model reliable, scalable, and accessible. The concept is straightforward, but implementing it correctly — especially at scale on Kubernetes — requires attention to many operational details. That’s what the rest of this course is designed to teach.
Next up in the course we’ll examine KServe’s architecture and how its components fit together on top of Kubernetes.Links and references
