Explains which AI serving behaviors run on CPUs versus which require high‑performance GPU fabric and validated benchmarks for Wide Expert Parallelism
This lesson separated two things: the architectural shapes and control-plane behavior you can observe on a small CPU cluster, and the GPU‑dependent performance characteristics that require a purpose‑built GPU fabric. Being explicit about where the line is helps you plan capacity, measure results, and avoid surprise rework.Wide Expert Parallelism is a GPU‑fleet feature — validated on 32 H200 or B200 GPUs connected over a high‑performance fabric. That is a purpose‑built environment, not a handful of accelerators tacked into a general-purpose cluster.
The all‑to‑all communication required for expert dispatch and the large block transfers for disaggregation rely on a low‑latency, high‑bandwidth fabric. In practice that means InfiniBand, RoCE, or cloud equivalents such as EFA — not ordinary Ethernet.
High-performance fabric is a requirement for Wide Expert Parallelism and model disaggregation. A standard data-center Ethernet network — and small development clusters — will not deliver the same latencies or throughput. See InfiniBand, RoCE, and EFA for details.
CPUs can run the same execution loop as an H200, which explains why the CPU‑based implementation reproduces system behavior. However, CPUs simply cannot match the throughput of a GPU fabric. The performance numbers quoted in this lesson are taken from the project’s published, reproducible benchmarks — use them as a baseline until you can measure your own environment.Key published results include throughput and throughput uplift numbers that were validated on the proper GPU fleet:
Benchmark tip: replicate the same workload, model, and network topology when you measure. Small differences in networking, GPU generation, or batch sizing will change throughput and latency numbers substantially.
Everything that does not require that specialized GPU fabric — and that list includes more than you might expect — ran live on our three‑pod cluster during this lesson. The control plane behaviors you saw are real and reproducible on CPUs:
Router scoring multiple pods and returning a duplicate request to an idle pod.
Cache hit counters incrementing for a single chosen replica.
Conversations growing turn by turn without backpressure at the control plane level.
The object model: objects, labels, pools, and profiles driving routing and capacity decisions.
The control plane itself is agnostic to the underlying chips. The same configuration and control-plane logic will work whether you point it at CPUs or a rack of GPUs — the difference is in speed, not in behavior.
Summary: the shapes and runtime behaviors are visible on CPUs; the speed claims come from GPU fabric benchmarks. Know which is which before you plan capacity or buy hardware.The practical takeaway is positive: GPUs were the only part of this story we had to narrate from a distance. Everything else — running a fleet, rolling out updates without dropping requests, monitoring dashboards, and validating with local benchmarks — is control‑plane work and runs on the three pods where it’s safe to break things and observe recovery.
Table: quick distinction between control plane work and GPU‑fabric requirements
The field trip to GPU land is over — you now know which parts you can validate locally and which parts need a GPU fabric to reach the published performance.
