Explains what a model server does, how vLLM loads models into GPU memory, impacts latency, startup time, scaling costs, and implications for concurrency and autoscaling.
In the previous lesson we left with a practical question: in the few seconds between sending a prompt and receiving an answer, what is the model server actually doing?This lesson opens up that “black box” and makes the model server concept concrete. Up to now, we’ve described vLLM as “wrapping the model in a service.” Here we’ll show what that means in practice and why it matters for latency, capacity planning, and cost.We previously ran a model from a short Python script that starts, loads the model, returns an answer, and exits. vLLM is the always-on version of that workflow: it starts once, loads the model into memory, and stays running to accept requests.To start a vLLM server you run a single command:
$ vllm serve Qwen/Qwen2.5-7B-InstructINFO Loading model weights from disk…
When you run this, vLLM reads the model weights — the billions of floating-point parameters — from disk and loads them into GPU memory (VRAM). These weight files are large (gigabytes), so startup is not instantaneous. Expect startup times from tens of seconds up to a few minutes depending on model size and hardware. This startup cost is important for capacity planning and responsiveness.Once the server is running, it listens for HTTP requests like any web service. One very useful detail: vLLM implements an OpenAI-compatible API format, so any client or tool that speaks the OpenAI API can talk to a local vLLM instance without code changes.
vLLM’s OpenAI-compatible API makes it straightforward to reuse existing clients and tooling. For example, point an OpenAI-style client at http://localhost:8000/v1 and it will work with a local vLLM server.
Two properties of this architecture heavily influence design choices when serving many users:
The server keeps the entire model loaded in GPU memory while it runs.
Starting the server requires time to load weights into VRAM.
These characteristics mean that scaling by adding servers is more expensive than scaling typical stateless web services. Each additional vLLM server requires its own copy of the model weights and a GPU with sufficient VRAM.
In short: a model server is heavy — expensive to run, slow to start, and costly to replicate. These constraints will shape how we design systems for concurrency, multi-tenancy, and autoscaling.Quick comparison: what running a model server means in practice
Characteristic
Implication
Example
Always-on model in VRAM
Low per-request inference latency once loaded, but high memory cost
vllm serve Qwen/...
Long startup time
Cannot instantly scale to handle bursts without pre-warmed capacity
Start time measured in tens of seconds to minutes
Cost to replicate
Each replica needs a GPU and a full copy of model weights
Running multiple vLLM instances per region increases cost
What happens during those few seconds between request and response?When a request arrives, the server:
Parses the incoming HTTP request and the prompt.
Runs the model’s forward pass on the GPU — token by token or in parallel, depending on the model and decoding strategy.
Streams or returns the generated tokens as the API response.
These steps map to real GPU computation: memory accesses to weights, matrix multiplications, attention operations, and token-decoding logic — all of which consume compute cycles and VRAM bandwidth.
Next, we’ll unpack these internal steps in more detail: how tokenization, attention, caching, and decoding shape latency; and what engineering approaches exist to improve throughput and reduce the cost of scaling model-serving infrastructure.Links and references
Remember: adding replicas to handle more traffic means additional GPUs and additional copies of the model weights. Plan capacity carefully and consider strategies like batching, model quantization, or multi-tenancy to reduce cost.
