> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# A Closer Look at the Model Server

> Explains what a model server does, how vLLM loads models into GPU memory, impacts latency, startup time, scaling costs, and implications for concurrency and autoscaling.

In the previous lesson we left with a practical question: in the few seconds between sending a prompt and receiving an answer, what is the model server actually doing?

This lesson opens up that “black box” and makes the model server concept concrete. Up to now, we've described vLLM as "wrapping the model in a service." Here we'll show what that means in practice and why it matters for latency, capacity planning, and cost.

We previously ran a model from a short Python script that starts, loads the model, returns an answer, and exits. vLLM is the always-on version of that workflow: it starts once, loads the model into memory, and stays running to accept requests.

To start a vLLM server you run a single command:

```bash theme={null}
$ vllm serve Qwen/Qwen2.5-7B-Instruct
INFO  Loading model weights from disk…
```

When you run this, vLLM reads the model weights — the billions of floating-point parameters — from disk and loads them into GPU memory (VRAM). These weight files are large (gigabytes), so startup is not instantaneous. Expect startup times from tens of seconds up to a few minutes depending on model size and hardware. This startup cost is important for capacity planning and responsiveness.

Once the server is running, it listens for HTTP requests like any web service. One very useful detail: vLLM implements an OpenAI-compatible API format, so any client or tool that speaks the OpenAI API can talk to a local vLLM instance without code changes.

vLLM's OpenAI-compatible API makes it straightforward to reuse existing clients and tooling. For example, point an `OpenAI`-style client at `http://localhost:8000/v1` and it will work with a local vLLM server.

A minimal HTTP request looks like this:

```http theme={null}
POST /v1/chat/completions HTTP/1.1
Host: localhost:8000
Content-Type: application/json

{
  "messages": [
    { "role": "user", "content": "Hi!" }
  ]
}
```

A typical (simplified) response from the server:

```json theme={null}
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "Hi! How can I help you today?"
      }
    }
  ]
}
```

You can also create an OpenAI-style client in Python that targets your local server:

```python theme={null}
# python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1")
```

Two properties of this architecture heavily influence design choices when serving many users:

* The server keeps the entire model loaded in GPU memory while it runs.
* Starting the server requires time to load weights into VRAM.

These characteristics mean that scaling by adding servers is more expensive than scaling typical stateless web services. Each additional vLLM server requires its own copy of the model weights and a GPU with sufficient VRAM.

![An infographic titled "A Second Server Is Not Cheap" showing two blue vLLM server blocks, a VRAM module, many small web app icons, and a dollar symbol. It visually conveys that spinning up another full server (with its own VRAM/weights) for many app copies is costly.](https://mintcdn.com/kodekloud-c4ac6d9a/HJ60thRsNzBTyswi/images/llm-d-A-Deep-Dive/From-ChatGPT-to-a-Model-Server/A-Closer-Look-at-the-Model-Server/vllm-second-server-not-cheap.jpg?fit=max\&auto=format\&n=HJ60thRsNzBTyswi\&q=85\&s=27cd02203821278b37c3582c51b9fa94)

In short: a model server is heavy — expensive to run, slow to start, and costly to replicate. These constraints will shape how we design systems for concurrency, multi-tenancy, and autoscaling.

Quick comparison: what running a model server means in practice

| Characteristic | Implication | Example |
| - | -: | - |
| Always-on model in VRAM | Low per-request inference latency once loaded, but high memory cost | `vllm serve Qwen/...` |
| Long startup time | Cannot instantly scale to handle bursts without pre-warmed capacity | Start time measured in tens of seconds to minutes |
| Cost to replicate | Each replica needs a GPU and a full copy of model weights | Running multiple vLLM instances per region increases cost |

What happens during those few seconds between request and response?

When a request arrives, the server:

1. Parses the incoming HTTP request and the prompt.
2. Runs the model’s forward pass on the GPU — token by token or in parallel, depending on the model and decoding strategy.
3. Applies decoding logic (sampling, beam search, temperature, top-k/top-p).
4. Streams or returns the generated tokens as the API response.

These steps map to real GPU computation: memory accesses to weights, matrix multiplications, attention operations, and token-decoding logic — all of which consume compute cycles and VRAM bandwidth.

![An infographic titled "Inside Those Few Seconds" showing a laptop labeled "You" sending requests to a central vLLM server that in turn communicates with a GPU (with VRAM and compute units). A 2.4 second interval is indicated between the components.](https://mintcdn.com/kodekloud-c4ac6d9a/HJ60thRsNzBTyswi/images/llm-d-A-Deep-Dive/From-ChatGPT-to-a-Model-Server/A-Closer-Look-at-the-Model-Server/inside-few-seconds-vllm-laptop-gpu.jpg?fit=max\&auto=format\&n=HJ60thRsNzBTyswi\&q=85\&s=26dd5e0db70c9584540923cc10644e5e)

Next, we'll unpack these internal steps in more detail: how tokenization, attention, caching, and decoding shape latency; and what engineering approaches exist to improve throughput and reduce the cost of scaling model-serving infrastructure.

Links and references

* vLLM: [https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)
* OpenAI API docs: [https://platform.openai.com/docs/api-reference](https://platform.openai.com/docs/api-reference)
* OpenAI Python client: [https://github.com/openai/openai-python](https://github.com/openai/openai-python)

Remember: adding replicas to handle more traffic means additional GPUs and additional copies of the model weights. Plan capacity carefully and consider strategies like batching, model quantization, or multi-tenancy to reduce cost.

> **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/b16615f2-d3fb-42ec-8823-f12ef7e820c6/lesson/5c31d96c-68e0-4130-84cd-1cc0483285e2)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.