> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Wide Expert Parallelism

> Explains why Mixture-of-Experts models benefit from expert parallelism, assigning whole experts per GPU, scaling wide and requiring RDMA-capable fabrics for efficient all-to-all communication

Earlier we discussed llm-d's "well-lit paths." One of those paths is Spread the Model — splitting a very large model across multiple GPUs when it cannot fit on a single device.

Two common ways to split a giant dense model are:

* Put whole layers on different GPUs (pipeline / layer parallelism).
* Split individual layers across GPUs (tensor / intra-layer parallelism).

Modern large models increasingly use a different strategy: Mixture-of-Experts (MoE). This article explains how MoE changes the best way to spread a model across devices and why "wide expert" deployments are often the right choice for MoE-based LLMs.

What is an MoE layer?

* A Mixture-of-Experts layer contains many independent expert subnetworks plus a small routing gate.
* The gate inspects each token and routes it to a small subset of experts; the remaining experts remain idle for that token.
* That design creates many specialist blocks where, for any given token, only a few experts are active.

![An infographic titled "One Famous Open Giant" shows a "DeepSeek R1" label above a grid of small rounded squares with several teal-highlighted cells. Arrows point from the grid to two large boxes displaying the numbers "256" and "8."](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Serving-Bigger-Models-With-llm-d/Wide-Expert-Parallelism/deepseek-r1-grid-arrows-256-8.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=ef608247e659dc1d42b784f9cf3a76c7)

For example, DeepSeek-R1 uses 256 routed experts per layer, with the router selecting 8 experts per token. That can produce models whose total parameter count is enormous, even though only a small fraction of weights are active for any single token.

![A dark infographic titled "Total vs Active" showing two horizontal progress bars: a long pink bar labeled "671 billion" and a much shorter teal bar labeled "37 billion." It visually compares a large total value against a much smaller active value.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Serving-Bigger-Models-With-llm-d/Wide-Expert-Parallelism/total-vs-active-progress-671b-37b.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=989bee11bfc51bb4e6903b55bab0ec5f)

In this example the model holds \~671B total parameters, but only \~37B are active for a given token (the chosen experts plus always-used parts of the model). The key observation: most weights sit idle for each token in an MoE design.

Why classic slicing is suboptimal for MoE

* In dense models, every token is processed by the same weights; slicing those weights (tensor or layer slicing) distributes those frequently-accessed pieces across GPUs efficiently.
* For MoE, slicing each expert across many GPUs fragments whole specialists. You end up with many GPUs holding tiny portions of many experts — most of which are unused for most tokens — and you still pay distribution and coordination costs.

![A dark-themed diagram titled "Slicing a Dense Giant" showing a long row of rounded blocks partitioned by dashed lines and each segment mapped down to four boxes labeled GPU 1 through GPU 4. It illustrates dividing a large computation or data block and distributing the slices across multiple GPUs.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Serving-Bigger-Models-With-llm-d/Wide-Expert-Parallelism/slicing-dense-giant-multi-gpu.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=01ba567b4a0150bdbc00d8e811351ae9)

Expert Parallelism: keep specialists whole

* Expert Parallelism assigns whole experts to individual GPUs. Rather than splitting a single expert across devices, each GPU owns several complete experts (e.g., GPU A holds experts 1–12, GPU B holds 13–24, etc.).
* This avoids fragmentation, reduces bookkeeping and cross-device assembly, and keeps each expert intact for efficient local execution.

![The image shows the title "Expert Parallelism" above two side-by-side panels labeled "GPU 1" and "GPU 2." Each panel contains a grid of rounded-square icons representing parallel processing slots or tasks.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Serving-Bigger-Models-With-llm-d/Wide-Expert-Parallelism/expert-parallelism-dual-gpu-panels.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=878e97c17712c744241e4e224460e8bb)

Scaling MoE: go wide

* MoE models still require hundreds of gigabytes for weights, so we scale them "wide" across many GPUs.
* The cluster exposes the distributed experts as a single logical model (one serving unit). Orchestration and health coordination use patterns like LeaderWorkerSet to keep the distributed experts functioning as a coherent service.

![A stylized infographic titled "Going Wide" showing a 100+ GB block distributing (dashed arrows) to four machines labeled Machine 1–4, each containing grouped container/pod icons. The diagram is labeled "LeaderWorkerSet" at the bottom.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Serving-Bigger-Models-With-llm-d/Wide-Expert-Parallelism/going-wide-100gb-leaderworkerset-machines.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=814307d3d9564a7e522a6b3a13539c03)

Benefits of Expert Parallelism

* No unnecessary replication of expert weights across GPUs.
* Freed device memory can be reallocated—for example, to increase KV cache capacity during generation.
* Simpler local execution for each expert reduces runtime overhead.

![A dark-themed diagram titled "Room for the KV Cache" showing four GPU boxes labeled GPU 1–4 (each with four small slots) across the top and a large VRAM bar below containing a "Model" block.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Serving-Bigger-Models-With-llm-d/Wide-Expert-Parallelism/kv-cache-room-gpus-vram-model.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=596756cc2a557ed43cd90bce39a78b9c)

All-to-all communication: the essential tradeoff

* When the router selects experts for tokens, tokens often need to be sent to experts on other machines, producing an everyone-to-everyone (all-to-all) communication pattern for each MoE layer.
* This pattern places a strong demand on the network fabric; it performs well only on RDMA-capable networks that support efficient GPU-to-GPU transfers.

![A dark-themed system diagram titled "Everyone to Everyone" showing a "Gate" on the left and three labeled machines (Machine 1, Machine 2, Machine 3), each containing two container-like groups of four small square pods. It appears to illustrate an all-to-all communication or deployment architecture.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Serving-Bigger-Models-With-llm-d/Wide-Expert-Parallelism/everyone-to-everyone-gate-machines.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=e937a1ac641fbed91ae41311f9425f75)

Fabrics built for MoE

* High-performance fabrics such as InfiniBand or RoCE are required in production-scale MoE deployments.
* Technologies like GPUDirect RDMA allow network adapters to move GPU memory across nodes without staging through host memory, which is crucial for low-latency, high-throughput all-to-all transfers.
* Conventional Ethernet-based datacenter networks typically cannot match this performance for large-scale MoE traffic patterns.

![A stylized diagram titled "Fabrics Built for It" showing three labeled server racks (Rack 1, Rack 2, Rack 3), each containing GPU icons. A connection bar below is labeled "InfiniBand or RoCE."](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Serving-Bigger-Models-With-llm-d/Wide-Expert-Parallelism/gpu-racks-infiniband-roce-fabric.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=bc0021ca3c068b104c65e0010eed4030)

For MoE serving at scale, plan for an RDMA-capable fabric (InfiniBand or RoCE). Without it, the all-to-all transfers between experts will be a major bottleneck.

Practical runtime optimization: prefill vs decode

* llm-d separates prefill (building the KV cache for past tokens) from decode (online token-by-token generation).
* Prefill is expensive but embarrassingly parallel across tokens; it benefits from wide expert-parallel deployments.
* Decode is latency-sensitive; splitting the phases reduces bandwidth pressure and latency during interactive, token-by-token generation while retaining the cost-effective scaling of wide deployments for the expensive prefill work.

Comparison: Parallelism strategies

| Parallelism Type | When to use | Pros | Cons |
| - | -: | - | - |
| Pipeline / Layer Parallelism | Large dense models with sequential layers | Good for streaming large layers across fewer GPUs | Requires synchronization across pipeline stages; limited parallelism when layers are imbalanced |
| Tensor / Intra-layer Parallelism | Very large dense layers that must be split | Fine-grained parallelism for matrix ops | All GPUs participate for every token; high inter-GPU bandwidth needs |
| Expert (Wide) Parallelism | MoE architectures with many experts | Keeps experts whole on single GPUs; reduces fragmentation and local bookkeeping; frees memory for KV cache | All-to-all communication between GPUs; requires RDMA-capable network |

Links and References

* [InfiniBand Trade Association](https://www.infinibandta.org/)
* [RoCE (RDMA over Converged Ethernet) Overview](https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet)
* [NVIDIA GPUDirect RDMA](https://developer.nvidia.com/gpudirect)
* For more on MoE architectures, see seminal papers and open-source MoE model releases for concrete examples and router designs.

Summary

* MoE models change the economics of model-parallel design: instead of slicing experts, give each GPU whole experts and scale wide.
* Expert Parallelism reduces weight replication and frees memory, but it trades those benefits for a heavier dependency on the network fabric.
* Plan cluster networks, GPU topology, and runtime phases (prefill vs decode) together for efficient, low-latency MoE serving.

> **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/4e52cd5b-6e1d-4d63-af4d-5442e6fa9b81/lesson/9d8afd53-7d60-4faf-963a-ded98faf3504)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.