> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Serving the Giants

> Describes sharding giant transformer models across multiple GPUs using pipeline and tensor parallelism, illustrating tradeoffs, communication patterns, and performance considerations for distributed inference.

Up to now we've pictured each server holding a complete model: one GPU, one copy of the weights, ready to answer. For many models that assumption still holds. But the very largest models break that assumption.

Some modern models are simply too large to fit on a single GPU. Model sizes vary widely: a small model can be a couple of gigabytes, a mid-sized model around 16 GB, a large model about 140 GB, and then there are the giants — models with hundreds of billions of parameters. For example, a 175B-parameter model can be on the order of \~350 GB in fp16, while 500B+ models may approach or exceed a terabyte on disk.

Now compare that to hardware. Even a top-end data-center GPU, the kind that costs as much as a car, typically holds about 80 GB of memory. The arithmetic is brutal: a model that needs several hundred gigabytes of weights versus a single GPU with \~80 GB — it doesn't fit.

You could buy the most expensive chip on the market,

![An infographic titled "The Arithmetic" comparing GPU memory, showing a top-end data-center GPU with 80 GB VRAM. A long bar labeled "several hundred GB" versus a small 80 GB block is shown with a "Not even close" note.](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/From-One-Server-to-a-Fleet/Serving-the-Giants/gpu-memory-80gb-vs-hundreds.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=2a0ad6a10727e5d2ba9f85ae3063b5ef)

and the giant still wouldn't load.

Solution: shard the model across multiple GPUs. Slice the giant into pieces and place a piece on each of many GPUs. No single GPU holds the whole model, but together they do. This is the essence of model parallelism for inference.

This partitioning is usually handled by model-serving software (for example, [vLLM](https://github.com/vllm-project/vllm)). To clients it still looks like one logical server: a single model endpoint at one address, while the actual work happens on a rack of chips coordinated by the serving layer.

```text theme={null}
http://llm.internal
```

A single logical endpoint (for example, `http://llm.internal`) can map to many GPUs. Model-serving frameworks hide the complexity of sharding so you keep a simple integration surface while benefiting from distributed inference.

![A stylized diagram titled "From the Outside" showing a laptop labeled "Your computer" connecting (via a URL) to a large "One Logical Server" box that contains multiple small model/container icons. It illustrates external access to a single logical server hosting many model instances.](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/From-One-Server-to-a-Fleet/Serving-the-Giants/external-access-one-logical-server-models.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=e3e0afb7df12173798aaeea9e1c1354a)

To decide how to slice the model, first picture the computation. A transformer is a stack of layers; each layer performs linear algebra (matrix multiplications, attention, etc.) and non-linearities. A prompt flows down the stack: the input enters at the top, each layer computes and passes activations down, and the final layer produces the next token.

Because the model computation is layered, there are two broad approaches to distributing it across GPUs:

* Pipeline parallelism (slice-by-layer): assign consecutive layers to different GPUs so the activation flows from GPU to GPU like an assembly line.
* Tensor (or model) parallelism: split each layer's computation across GPUs, with each GPU computing partial results that are combined to form the full layer output.

Pipeline parallelism is a slice-by-layer approach: give the first group of layers to GPU 1, the next group to GPU 2, and so on. The computation flows down GPU 1's layers, then is handed off to GPU 2 — relatively few handoffs but each handoff moves a larger activation.

![A diagram titled "By Layer" showing two GPU boxes (GPU 1 and GPU 2), each containing stacked neural network layers (L1–L3 on GPU 1 and L4–L6 on GPU 2). A large green arrow points from GPU 1 toward GPU 2, indicating layer-wise distribution or transfer.](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/From-One-Server-to-a-Fleet/Serving-the-Giants/layerwise-transfer-gpu1-to-gpu2.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=6147910f6403010ad73d11c73475ea9c)

Tensor (or within-layer) parallelism splits the computation inside each layer: each GPU computes part of a layer's terms and the partial results are combined at every layer. This approach increases communication frequency because GPUs exchange partial results for each layer, but it can help when individual matrix multiplications themselves exceed a single GPU's capacity.

Either way, note what actually travels between GPUs: small intermediate activations or partial results — not the full weight tensors. The model weights are loaded once on their assigned GPU and remain resident in local GPU memory.

![A diagram titled "What Travels" showing two GPUs (GPU1 and GPU2), each with a grid of small tiles and a VRAM column, connected by a link labeled "Small results" to indicate the data exchanged between them.](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/From-One-Server-to-a-Fleet/Serving-the-Giants/gpu-to-gpu-small-results.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=cf983454561f457cdac07e787b92ea4a)

Keeping weights local helps with the autoregressive decoding bottleneck: each new token triggers another forward pass, and when weights are sharded each GPU reads only its local slice of weights from local memory. That reduces per-GPU memory bandwidth pressure and can enable very high token throughput when the cluster and networking are well-configured.

Two common trade-offs when choosing a sharding strategy:

| Strategy | Partition axis | Communication pattern | Pros | Cons |
| - | - | - | - | - |
| Pipeline parallelism | By layer (consecutive layers per GPU) | Fewer, larger handoffs between GPUs | Simpler scheduling; lower per-layer communication frequency | Larger activation transfers at handoffs; potential pipeline bubbles |
| Tensor (model) parallelism | Within each layer | Frequent per-layer exchanges of partial results | Can split large matrix ops; balances compute across GPUs within layer | Higher communication overhead and stricter latency requirements |

![A slide titled "Two Ways to Slice" showing two rounded pink gradient blocks side by side labeled "By Layer" (with a dashed horizontal cut) and "Within Each Layer" (with a dashed vertical cut). A small question-mark icon sits between the two blocks.](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/From-One-Server-to-a-Fleet/Serving-the-Giants/slice-by-layer-vs-within-layer.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=e9b087612f18ad6af4dc7e5a0a239c5f)

Which approach to use depends on where your system's bottlenecks are: network bandwidth and latency, per-GPU memory capacity, and the specific shapes of the model's matrices. A well-sharded cluster—tuned for the model and the workload—can serve thousands to tens of thousands of tokens per second, but poor choices around sharding and interconnects will limit throughput.

Choosing the wrong sharding strategy for your hardware can hurt performance. High-frequency communication (tensor parallelism) demands low-latency, high-bandwidth interconnects; pipeline parallelism may require careful scheduling to avoid pipeline stalls. Profile your model and network before committing to a strategy.

References and further reading

* vLLM — a model-serving framework that supports sharded inference: [https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)
* Transformer architecture overview: [https://arxiv.org/abs/1706.03762](https://arxiv.org/abs/1706.03762)

> **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/5ec7ad88-5c69-4985-a776-3f7bca88fbf3/lesson/5ad0a9dd-0def-48f0-8177-48adc8b3dcb4)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.