Skip to main content
Earlier we discussed llm-d’s “well-lit paths.” One of those paths is Spread the Model — splitting a very large model across multiple GPUs when it cannot fit on a single device. Two common ways to split a giant dense model are:
  • Put whole layers on different GPUs (pipeline / layer parallelism).
  • Split individual layers across GPUs (tensor / intra-layer parallelism).
Modern large models increasingly use a different strategy: Mixture-of-Experts (MoE). This article explains how MoE changes the best way to spread a model across devices and why “wide expert” deployments are often the right choice for MoE-based LLMs. What is an MoE layer?
  • A Mixture-of-Experts layer contains many independent expert subnetworks plus a small routing gate.
  • The gate inspects each token and routes it to a small subset of experts; the remaining experts remain idle for that token.
  • That design creates many specialist blocks where, for any given token, only a few experts are active.
An infographic titled "One Famous Open Giant" shows a "DeepSeek R1" label above a grid of small rounded squares with several teal-highlighted cells. Arrows point from the grid to two large boxes displaying the numbers "256" and "8."
For example, DeepSeek-R1 uses 256 routed experts per layer, with the router selecting 8 experts per token. That can produce models whose total parameter count is enormous, even though only a small fraction of weights are active for any single token.
A dark infographic titled "Total vs Active" showing two horizontal progress bars: a long pink bar labeled "671 billion" and a much shorter teal bar labeled "37 billion." It visually compares a large total value against a much smaller active value.
In this example the model holds ~671B total parameters, but only ~37B are active for a given token (the chosen experts plus always-used parts of the model). The key observation: most weights sit idle for each token in an MoE design. Why classic slicing is suboptimal for MoE
  • In dense models, every token is processed by the same weights; slicing those weights (tensor or layer slicing) distributes those frequently-accessed pieces across GPUs efficiently.
  • For MoE, slicing each expert across many GPUs fragments whole specialists. You end up with many GPUs holding tiny portions of many experts — most of which are unused for most tokens — and you still pay distribution and coordination costs.
A dark-themed diagram titled "Slicing a Dense Giant" showing a long row of rounded blocks partitioned by dashed lines and each segment mapped down to four boxes labeled GPU 1 through GPU 4. It illustrates dividing a large computation or data block and distributing the slices across multiple GPUs.
Expert Parallelism: keep specialists whole
  • Expert Parallelism assigns whole experts to individual GPUs. Rather than splitting a single expert across devices, each GPU owns several complete experts (e.g., GPU A holds experts 1–12, GPU B holds 13–24, etc.).
  • This avoids fragmentation, reduces bookkeeping and cross-device assembly, and keeps each expert intact for efficient local execution.
The image shows the title "Expert Parallelism" above two side-by-side panels labeled "GPU 1" and "GPU 2." Each panel contains a grid of rounded-square icons representing parallel processing slots or tasks.
Scaling MoE: go wide
  • MoE models still require hundreds of gigabytes for weights, so we scale them “wide” across many GPUs.
  • The cluster exposes the distributed experts as a single logical model (one serving unit). Orchestration and health coordination use patterns like LeaderWorkerSet to keep the distributed experts functioning as a coherent service.
A stylized infographic titled "Going Wide" showing a 100+ GB block distributing (dashed arrows) to four machines labeled Machine 1–4, each containing grouped container/pod icons. The diagram is labeled "LeaderWorkerSet" at the bottom.
Benefits of Expert Parallelism
  • No unnecessary replication of expert weights across GPUs.
  • Freed device memory can be reallocated—for example, to increase KV cache capacity during generation.
  • Simpler local execution for each expert reduces runtime overhead.
A dark-themed diagram titled "Room for the KV Cache" showing four GPU boxes labeled GPU 1–4 (each with four small slots) across the top and a large VRAM bar below containing a "Model" block.
All-to-all communication: the essential tradeoff
  • When the router selects experts for tokens, tokens often need to be sent to experts on other machines, producing an everyone-to-everyone (all-to-all) communication pattern for each MoE layer.
  • This pattern places a strong demand on the network fabric; it performs well only on RDMA-capable networks that support efficient GPU-to-GPU transfers.
A dark-themed system diagram titled "Everyone to Everyone" showing a "Gate" on the left and three labeled machines (Machine 1, Machine 2, Machine 3), each containing two container-like groups of four small square pods. It appears to illustrate an all-to-all communication or deployment architecture.
Fabrics built for MoE
  • High-performance fabrics such as InfiniBand or RoCE are required in production-scale MoE deployments.
  • Technologies like GPUDirect RDMA allow network adapters to move GPU memory across nodes without staging through host memory, which is crucial for low-latency, high-throughput all-to-all transfers.
  • Conventional Ethernet-based datacenter networks typically cannot match this performance for large-scale MoE traffic patterns.
A stylized diagram titled "Fabrics Built for It" showing three labeled server racks (Rack 1, Rack 2, Rack 3), each containing GPU icons. A connection bar below is labeled "InfiniBand or RoCE."
For MoE serving at scale, plan for an RDMA-capable fabric (InfiniBand or RoCE). Without it, the all-to-all transfers between experts will be a major bottleneck.
Practical runtime optimization: prefill vs decode
  • llm-d separates prefill (building the KV cache for past tokens) from decode (online token-by-token generation).
  • Prefill is expensive but embarrassingly parallel across tokens; it benefits from wide expert-parallel deployments.
  • Decode is latency-sensitive; splitting the phases reduces bandwidth pressure and latency during interactive, token-by-token generation while retaining the cost-effective scaling of wide deployments for the expensive prefill work.
Comparison: Parallelism strategies Links and References Summary
  • MoE models change the economics of model-parallel design: instead of slicing experts, give each GPU whole experts and scale wide.
  • Expert Parallelism reduces weight replication and frees memory, but it trades those benefits for a heavier dependency on the network fabric.
  • Plan cluster networks, GPU topology, and runtime phases (prefill vs decode) together for efficient, low-latency MoE serving.

Watch Video