Skip to main content
Last time we looked at how to run models that are too large for a single GPU by slicing them across multiple GPUs. There are two common slicing strategies:
  • Slice by layer (pipeline parallelism): give each device a contiguous set of layers.
  • Slice within a layer (tensor/model parallelism): split the math inside each layer across GPUs.
These two approaches — and how you combine them — are collectively known as model sharding. Which to choose depends mainly on where the GPUs live and, critically, how fast they can communicate.
A slide titled "Two Ways to Slice" showing two pink rounded rectangles on a dark background: the left illustrates a dashed horizontal cut labeled "By Layer" with a green arrow and note "Quiet — one handoff," and the right shows a dashed vertical cut labeled "Within Each Layer."
Where GPUs are placed changes the trade-offs. If GPUs are in different machines, communication goes over the network; if they’re in the same chassis, they can use a high-speed, low-latency interconnect.
A dark-themed diagram titled "Where the GPUs Live" showing two stylized GPU boxes labeled "GPU 1" (left) and "GPU 2" (right). They are linked by a horizontal line with a red question-mark in the center, indicating an unknown or missing connection.
Inside a single machine, GPUs are connected by NVLink (NVIDIA) or xGMI (AMD). These interconnects deliver far higher bandwidth and lower latency than Ethernet or even typical InfiniBand networks, which makes the “chatty” approach — splitting a layer’s computation across GPUs — practical.
An infographic titled "Inside One Machine" showing a stylized machine rack with eight small node panels arranged in two rows and linked by a central green bus. The graphic uses a blue/teal color scheme with rounded module icons.
On fast intra-node links you can afford frequent exchanges of intermediate tensors (activations, gradients, etc.), so tensor (within-layer) parallelism performs well. Put eight NVLink-connected GPUs in one chassis and they often operate like one logical device — this pattern is how many large models are trained and served today.
An infographic titled "Eight GPUs, One Box" shows eight small GPU module icons arranged inside a single chassis and linked by a central horizontal interconnect. A caption below reads "How most large models run today" and a green checkmark appears at the side.
When you span multiple machines, the network is usually slower and higher-latency than NVLink/xGMI. Across-node traffic should therefore be minimized: pipeline parallelism (splitting by layers) sends a single activation from one machine to the next, producing relatively light network load that conventional inter-node links can sustain.
An infographic titled "The Light Method" with a central pill button labeled "Light – whole stretches." Below it are two rounded boxes labeled "Machine 1" and "Machine 2" containing stacked list items (L1–L3 and L4–L6) connected by a dotted line and a "Little traffic" label.
The simple rule of thumb is: keep the chatty (high-bandwidth, low-latency) traffic inside a machine, and keep only light traffic across machines.
Sharding chooses between tensor/model parallelism (within a machine) and pipeline parallelism (across machines) based on interconnect speed. The goal is to minimize slow, high-volume network transfers.
Quick comparison — when to use each sharding method: vLLM provides a practical way to describe and launch these configurations. A single command starts a server that can apply intra-node (chatty) and inter-node (light) splitting according to the parallelism settings:
You can configure tensor- and pipeline-parallel sizes when you launch. Example:
Check the vLLM documentation for exact flag names and supported combinations for your version.
An infographic titled "The Fitting Problem, Solved" with a pink "Giant" banner at the top. It shows two grouped boxes of small server icons illustrating a single machine's worth versus several machines' worth mapped to one logical server.
Sharding solves the weight-placement (fitting) problem: by spreading parameters across GPUs you can host models that would not fit on a single device. But sharding does not increase the runtime memory available to a single server session — activation buffers, per-request scratch space, and KV caches still consume each server’s local RAM/VRAM. For production services you therefore need a fleet of servers: each server hosts the model (sharded across its GPUs) and serves a subset of users. The architecture then raises a classic routing question: where should a user’s next message be sent?
A stylized diagram titled "A Whole Fleet" showing four servers (Server 1–4), each hosting a pink "Model" box. A KV cache icon is shown above linked to one server, with small user/client icons beneath each server.
A user’s KV cache — the cached activations and attention keys that make a chat respond quickly — lives in the memory of the server that created it. If the user’s next message is routed to the same server, the reply is fast. If it lands on a different server, that server must reconstruct the conversation state (or fetch it over the network), increasing latency.
In multi-server deployments, incorrect routing or misplaced caches causes recomputation or cache transfers, which degrade latency and throughput. Design routing and cache placement carefully.
Routing and cache management are therefore essential for low-latency, high-throughput serving. Common solutions include sticky routing (send a user to the same server while their KV cache is hot), sharded KV layers, or a shared fast cache layer — each has trade-offs in complexity, memory usage, and network load. Links and references

Watch Video