Explains why GPUs with VRAM are needed for large models, focusing on compute, memory bandwidth, and capacity tradeoffs.
We begin with a puzzle.A model is just a big file, and running it is just loading that file into memory and doing lots of math.So why can’t the perfectly good computer in front of you run one?
To answer that, we need to look inside your computer.Think about any computer you use — laptop or data-center server. Inside there are two parts we care about most:
The CPU (processor): the part that does the thinking — the calculations, logic, and orchestration.
The RAM (system memory): where the computer keeps the data the CPU is currently working on.
Because data is needed later, these two parts usually sit separately. Take the smallest calculation there is: 6 times 7. Those two numbers land in memory first. The CPU reads them out, does the multiplication, and writes 42 back. Every calculation is the same round trip: compute ↔ memory.
Processor here. Memory there. A road running between them. Hold that picture.The CPU is a brilliant generalist: a handful of very powerful cores that work through complicated tasks, one after another, very fast. For almost everything a general-purpose computer does, that’s exactly what you want.What does a model actually do under the hood? At its core, running a model is billions of very simple multiplications. Crucially, most of those multiplications are independent — they could happen at the same time.Hand those operations to a CPU and its few powerful cores must work through those billions of little multiplications mostly in sequence, a few at a time. It gets every one right, but you’ll be waiting a very, very long time.This is the first problem: the work is massively parallel, but the CPU is built for sequences.
We need a different kind of processor. Enter the GPU.A GPU is built the opposite way to a CPU: instead of a handful of powerful cores, it has thousands of small, simple cores designed to do the same operation across many data items at once. A modest data-center GPU like the NVIDIA T4 has roughly 2,500 of these small cores. Where a CPU might manage up to billions or a few trillion operations per second on some workloads, a GPU can perform many trillions of operations per second — often one to two orders of magnitude more math throughput for highly parallel workloads. The mountain of multiplications that crawled on the CPU now gets done in a flash.So the GPU solves our compute problem, but it creates a new one. A processor needs memory to feed it. Now we’ve got thousands of hungry cores, and every one of them needs the model’s weights — all those billions of numbers delivered constantly:
Where do the weights live? Leave them in system RAM and we’ve got a bandwidth problem. System RAM is outside the GPU; the GPU must fetch data from it over interconnects (PCIe or NVLink). Typical CPU–GPU interconnects provide bandwidth in the tens of gigabytes per second, much lower than the GPU’s on-board memory bandwidth which is in the hundreds of gigabytes per second. For thousands of cores all demanding data at once, that off-board memory often can’t deliver fast enough — the cores sit idle, waiting for numbers that arrive too slowly, and all that parallel power is starved.
How do we feed thousands of cores fast enough? The answer is to give the GPU its own memory, built onto the same card right next to its cores. This is VRAM.
Now the model’s weights can live right beside the math, and the cores can be fed over a much wider pipe. VRAM bandwidth is measured in hundreds of gigabytes to terabytes per second depending on the card — orders of magnitude more than system RAM bandwidth.
But VRAM has a trade-off: it’s fast but limited in capacity. A card like the T4 has 16 GB of VRAM. For example, a large, 70 billion parameter model can need roughly 140 GB just to hold its weights in memory (assuming about 2 bytes per parameter). That’s far larger than what’s available on a single 16 GB card.
Space in VRAM is precious. Remember this because it becomes a big deal later.Put these observations together and you get three numbers that size every AI workload:
Dimension
What it measures
Why it matters
Compute
How much math the card can do (number of cores × frequency)
Determines raw throughput — how many operations per second the accelerator can execute.
Bandwidth
How fast the GPU can read/write its own memory (VRAM)
Determines whether the cores are starved or fully utilized when reading weights and activations.
Capacity
How much fits in the GPU’s memory at all (VRAM size)
Limits which models fit on a single card and influences the need for model partitioning or offloading.
Every accelerator is described by those three numbers. The card you pick decides which models fit and how fast they will run. Bandwidth in particular is important: when we later watch a model actually generate text, that reading speed will determine a lot of the runtime behavior.
Three things to remember about any GPU for models: compute, bandwidth, and capacity. A fast GPU with too little VRAM still can’t hold a large model; plenty of VRAM with low bandwidth still starves the cores.
So let’s take stock. We’ve got a GPU with thousands of cores and its own memory sitting right beside them. The hardware looks ready — at least it looks ready — because as we’ll see, there are more complications that affect how models actually run efficiently.
