Explains turning a PyTorch model script into an always-on GPU model server with vLLM providing batching, scheduling, memory management and OpenAI compatible API
Last time we prepared the hardware: a GPU with thousands of compute cores and high-bandwidth video memory (VRAM) ready to hold a model’s weights.So, are we done?Not quite.
A chip alone is just potential. That GPU can perform huge amounts of math very quickly, but something must:
load the model weights into VRAM,
hand data and work to the GPU cores,
and collect the results to return to the requester.
Silicon doesn’t orchestrate this by itself — we need software.The standard tooling for that orchestration is PyTorch. With simple, idiomatic Python you describe tensor operations and model logic, and PyTorch handles device transfer and GPU execution for you.Minimal PyTorch example:
import torchx = torch.randn(4, 4, device="cuda")y = x @ x
To run a full language model you typically combine PyTorch with the Hugging Face Transformers library, which provides model definitions and tokenizers. Under the hood Transformers uses PyTorch (or other backends), so model execution happens on the GPU.Example: load a model, tokenize input, and generate text:
from transformers import AutoModelForCausalLM, AutoTokenizerimport torchmodel_name = "gpt2"tokenizer = AutoTokenizer.from_pretrained(model_name)model = AutoModelForCausalLM.from_pretrained(model_name).to("cuda")inputs = tokenizer("Hello, how are you?", return_tensors="pt").to("cuda")out = model.generate(**inputs, max_new_tokens=40)print(tokenizer.decode(out[0], skip_special_tokens=True))
At this level you already have everything needed to run the model locally. But two practical problems remain.
Networking and availability: that script runs once on your machine and then exits. Real services must run continuously and accept requests from many users across the internet.
Efficiency and cost: a single generate call handles one request end-to-end:
out = model.generate(**inputs, max_new_tokens=40)
GPUs are expensive and billed by time. Running a single request at a time leaves the GPU under-utilized. Keeping the GPU busy and serving many users concurrently requires scheduling, batching, and memory management that go beyond a simple script.
Production serving is not just about correctness — it’s about latency, throughput, and cost. A naive one-request-per-run approach can balloon cloud costs quickly.
A simple script might be represented conceptually as:
# run.py (conceptual)# starts, runs a single request, then exits# production needs a process that stays up and accepts network requests
We need a component that does both: turns a model into an always-on network service and runs it efficiently so the hardware is cost-effective. That component is the model server. One widely used example is vLLM.vLLM is an open-source model server built on top of PyTorch. It wraps the model, keeps it resident in GPU memory, and exposes a web API — often using an OpenAI-style surface so many existing clients and tools can talk to it without changes.
vLLM exposes OpenAI-compatible endpoints so you can often reuse existing clients, SDKs, and tooling that already support that API.
That solves the always-on networking problem. The other half — efficiency — is vLLM’s central purpose. Under the friendly API, vLLM implements advanced scheduling, request batching, and memory-aware placement so a single GPU can serve many concurrent users without wasting capacity.What does this buy you in practice? Here’s a quick comparison.
Concern
Script (single-run)
Model Server (vLLM-style)
Availability
One-off, local runs
Always-on HTTP API
Concurrency
Single request per run
Many concurrent requests with scheduling
GPU utilization
Often under-utilized
Higher utilization via batching and scheduling
Tooling compatibility
Manual integration
OpenAI-style API compatibility
Production readiness
Requires glue code for scaling
Built-in memory and request management
So now a real product exists: people on the internet can send requests and receive answers. That brings us to the next question: what exactly does the model server do internally for each incoming request? How does it break down work across GPU cores, when does it batch requests, and how does it manage memory footprints and prompt context?We’ll explore that machinery next.
