> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# From a Script to a Service

> Explains turning a PyTorch model script into an always-on GPU model server with vLLM providing batching, scheduling, memory management and OpenAI compatible API

Last time we prepared the hardware: a GPU with thousands of compute cores and high-bandwidth video memory (VRAM) ready to hold a model's weights.

So, are we done?

Not quite.

![A stylized infographic titled "Is the Hardware Ready?" showing a GPU diagram with a grid of blue squares labeled "10,000 cores" next to a large block labeled "VRAM" (fast memory). It highlights GPU cores and memory as key hardware components.](https://mintcdn.com/kodekloud-c4ac6d9a/HJ60thRsNzBTyswi/images/llm-d-A-Deep-Dive/From-ChatGPT-to-a-Model-Server/From-a-Script-to-a-Service/gpu-cores-vram-infographic.jpg?fit=max\&auto=format\&n=HJ60thRsNzBTyswi\&q=85\&s=19c186f4e0ee57a39f027c20c8bc009d)

A chip alone is just potential. That GPU can perform huge amounts of math very quickly, but something must:

* load the model weights into VRAM,
* hand data and work to the GPU cores,
* and collect the results to return to the requester.

Silicon doesn't orchestrate this by itself — we need software.

The standard tooling for that orchestration is PyTorch. With simple, idiomatic Python you describe tensor operations and model logic, and PyTorch handles device transfer and GPU execution for you.

Minimal PyTorch example:

```python theme={null}
import torch

x = torch.randn(4, 4, device="cuda")
y = x @ x
```

To run a full language model you typically combine PyTorch with the Hugging Face Transformers library, which provides model definitions and tokenizers. Under the hood Transformers uses PyTorch (or other backends), so model execution happens on the GPU.

Example: load a model, tokenize input, and generate text:

```python theme={null}
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name).to("cuda")

inputs = tokenizer("Hello, how are you?", return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

At this level you already have everything needed to run the model locally. But two practical problems remain.

1. Networking and availability: that script runs once on your machine and then exits. Real services must run continuously and accept requests from many users across the internet.

2. Efficiency and cost: a single generate call handles one request end-to-end:

```python theme={null}
out = model.generate(**inputs, max_new_tokens=40)
```

GPUs are expensive and billed by time. Running a single request at a time leaves the GPU under-utilized. Keeping the GPU busy and serving many users concurrently requires scheduling, batching, and memory management that go beyond a simple script.

Production serving is not just about correctness — it's about latency, throughput, and cost. A naive one-request-per-run approach can balloon cloud costs quickly.

A simple script might be represented conceptually as:

```bash theme={null}
# run.py (conceptual)
# starts, runs a single request, then exits
# production needs a process that stays up and accepts network requests
```

We need a component that does both: turns a model into an always-on network service and runs it efficiently so the hardware is cost-effective. That component is the model server. One widely used example is vLLM.

vLLM is an open-source model server built on top of PyTorch. It wraps the model, keeps it resident in GPU memory, and exposes a web API — often using an OpenAI-style surface so many existing clients and tools can talk to it without changes.

![A stylized diagram titled "vLLM Wraps the Model" showing a stack of components (Model, PyTorch, GPU) on the left connected via an API endpoint (/v1/chat/completions) to an OpenAI-style interface. On the right are icons for terminal, chat, and code interfaces, with a small "Running" status indicator.](https://mintcdn.com/kodekloud-c4ac6d9a/HJ60thRsNzBTyswi/images/llm-d-A-Deep-Dive/From-ChatGPT-to-a-Model-Server/From-a-Script-to-a-Service/vllm-wraps-model-openai-api-diagram.jpg?fit=max\&auto=format\&n=HJ60thRsNzBTyswi\&q=85\&s=d1a4bdde39e7a8fc063908d9270a4c6f)

vLLM exposes OpenAI-compatible endpoints so you can often reuse existing clients, SDKs, and tooling that already support that API.

That solves the always-on networking problem. The other half — efficiency — is vLLM’s central purpose. Under the friendly API, vLLM implements advanced scheduling, request batching, and memory-aware placement so a single GPU can serve many concurrent users without wasting capacity.

What does this buy you in practice? Here’s a quick comparison.

| Concern | Script (single-run) | Model Server (vLLM-style) |
| -: | - | - |
| Availability | One-off, local runs | Always-on HTTP API |
| Concurrency | Single request per run | Many concurrent requests with scheduling |
| GPU utilization | Often under-utilized | Higher utilization via batching and scheduling |
| Tooling compatibility | Manual integration | OpenAI-style API compatibility |
| Production readiness | Requires glue code for scaling | Built-in memory and request management |

So now a real product exists: people on the internet can send requests and receive answers. That brings us to the next question: what exactly does the model server do internally for each incoming request? How does it break down work across GPU cores, when does it batch requests, and how does it manage memory footprints and prompt context?

We’ll explore that machinery next.

![A stylized architecture diagram titled "A Real Product" showing many user icons sending requests into a central blue box labeled "vLLM," which then connects to a GPU block with compute units and VRAM.](https://mintlify.s3.us-west-1.amazonaws.com/kodekloud-c4ac6d9a/images/llm-d-A-Deep-Dive/From-ChatGPT-to-a-Model-Server/From-a-Script-to-a-Model-Server/vllm-gpu-architecture-users-requests.jpg)

References and further reading

* PyTorch: [https://pytorch.org](https://pytorch.org)
* Hugging Face Transformers: [https://huggingface.co/docs/transformers/index](https://huggingface.co/docs/transformers/index)
* vLLM project: [https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)
* OpenAI API reference: [https://platform.openai.com/docs/api-reference](https://platform.openai.com/docs/api-reference)

> **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/b16615f2-d3fb-42ec-8823-f12ef7e820c6/lesson/59f57a0a-a6dc-4690-9944-1cc59ce4c86d)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.