Skip to main content
Welcome back! A model server is the bridge between a trained model and a live system that uses it. KServe brings that bridge to Kubernetes by adding ML-aware abstractions on top of Kubernetes primitives. This article explains KServe’s architecture, how its control and data planes interact, and the essential resources you’ll use to serve models reliably. KServe is Kubernetes-native: it extends Kubernetes with Custom Resource Definitions (CRDs) that represent model-serving concepts. The primary CRD you will work with is the InferenceService. The InferenceService is the main declarative abstraction. You provide a name, the model format, and a model location — for example s3://my-bucket/my-model, gs://my-bucket/my-model, or a Hugging Face model ID. KServe handles provisioning, runtime selection, networking, and status reporting based on that manifest.
The image illustrates a flow diagram featuring KServe and InferenceService, showcasing their components and relationships, including Custom Resource Definitions and various storage-related attributes.
When you apply an InferenceService YAML to the cluster, the KServe controller-manager observes it and begins reconciliation. The controller-manager is the control plane: it watches InferenceService resources, picks a serving runtime, creates the underlying compute resources, configures networking, and updates resource status (for example indicating readiness and the service URL).
The image is a diagram labeled "The Control Plane," showing connections between InferenceService, Cluster, Kserve Controller Manager, and components like Serving Runtime and Networking.
This follows Kubernetes’ reconciliation pattern: declare the desired state, and the control plane enforces it. Change the model URI and the controller will roll out a new version. Restart the cluster and the controller will restore the declared state. KServe does not provide a single monolithic model server. Instead it uses ClusterServingRuntimes — templates that map model formats to container images and pod specifications. KServe ships with runtimes for common frameworks:
  • scikit-learn
  • XGBoost
  • LightGBM
  • PyTorch (via TorchServe)
  • NVIDIA Triton
  • Hugging Face servers for generative/predictive models
You can also add custom ClusterServingRuntimes for frameworks or deployment patterns that aren’t included out of the box. When you declare a model format in an InferenceService, the controller chooses the appropriate runtime and constructs the Pod spec.
The image displays a list of serving runtimes for machine learning, including Scikit-learn, XGBoost, LightGBM, and others, under the category "ClusterServingRuntimes." It also notes that the runtime matches the pod specification.
The control plane handles lifecycle and orchestration; the data plane handles live inference traffic. A prediction request follows this flow: it arrives at an ingress point, is routed to the predictor component, the serving runtime loads the model, runs inference, and returns the result. KServe supports an optional transformer component for pre-processing and post-processing. Placeholders for protocol translation, input normalization, or response formatting are implemented in the transformer so you can keep the serving runtime focused on inference.
Use transformers to separate preprocessing/postprocessing from the predictor. This keeps model containers simple and lets you update request/response logic independently of the model server.
The image illustrates a data flow chart titled "The Data Plane – Request Flow," showing steps like "Ingress Point," "Route to Predictor," "Run Inference," and "Return Result."
A key implementation detail is the storage initializer. When the predictor Pod is created, KServe injects an init container that runs before the model server starts. This initializer downloads the model artifact from the configured storage backend — S3, GCS, Azure Blob, a PVC, or Hugging Face — and places it at a known local path inside the Pod. The model server then reads the artifact from that local path. This abstraction hides the artifact’s origin from the model server; the runtime only needs to read a local path, so your InferenceService can point to different storage backends interchangeably.
The image is a flowchart titled "The Storage Initializer," describing the process where Kserve creates a pod, injects an init container, which downloads a model artifact and places it in a known local path. Various icons are displayed at the bottom.
Deployment modes KServe supports two main deployment modes. Choose based on resource cost, operational complexity, and latency requirements:
Serverless mode saves resources by scaling idle models to zero, but expect cold-start latency on the first request and additional components to operate.
The image illustrates deployment modes, focusing on a "Serverless" option with features like scaling to zero pods, freeing GPU/CPU resources, scale backup, and operational complexity challenges. A "RawDeployment" option is also mentioned.
These examples use RawDeployment because it’s easier to run locally with tools like Minikube or kind, and it keeps the focus on KServe itself rather than serverless components.
The image compares "Deployment Modes" with selections for "Serverless" and "RawDeployment," highlighting features such as "Simple," "No Cold Start," and "No Scale to Zero." It includes icons for Kubernetes and related tools.
Putting it all together: the typical lifecycle
  1. Write an InferenceService manifest (name, model format, model URI).
  2. Apply it to the cluster.
  3. KServe controller-manager detects the resource, selects a ClusterServingRuntime, and creates Pods.
  4. Each Pod gets a storage initializer that downloads the model to a local path.
  5. The model server loads the artifact and begins serving.
  6. Ingress routes requests to the predictor (and optional transformer), and predictions are returned to clients.
KServe gives you a declarative, Kubernetes-native way to automate the hard parts of model-serving infrastructure: runtime selection, artifact provisioning, scaling, and networking.
The image is a flowchart titled "Putting It All Together," outlining steps in a process involving writing an InferenceService, a controller picking it up, selecting a ClusterServingRuntime, downloading a model, and serving it.
Next steps This architecture can feel abstract until you use actual resources. To get hands-on, set up a local cluster and install KServe: References and further reading In the following lessons we’ll walk through installing KServe on a local cluster and deploying a sample InferenceService so you can experience the entire control/data plane lifecycle end-to-end.

Watch Video