What HBM memory is, and why AI chips can never get enough of it

1,024bit-wide data path on a single HBM3E stack

What to know

HBM is ordinary DRAM made into a vertical stack of chips, wired together and placed right next to the processor.
Its very wide connection moves far more data per second than the memory in a PC or a graphics card.
It takes more silicon per bit, fails more often in production and needs special packaging, so supply grows slowly.

Every AI accelerator sold for data centers, from Nvidia's and AMD's to the custom chips the cloud companies design, has the same thing sitting beside its processor: a few small towers of memory called HBM, short for high-bandwidth memory. It is one of the most expensive parts of the package and often the part in shortest supply.

What it is

HBM is DRAM, the same kind of working memory found in a laptop. The difference is how it is built and where it sits.

A normal memory module spreads chips flat across a circuit board, several centimeters from the processor. HBM stacks thin DRAM dies on top of each other, commonly 8, 12 or 16 high, on a base chip that manages them. Tiny vertical connections drilled through the silicon, called through-silicon vias, carry signals up and down the stack.

The finished stack is then placed in the same package as the processor, a few millimeters away, on a layer of silicon that carries thousands of fine wires between them. This arrangement is often called 2.5D packaging. TSMC's version, CoWoS, is the best-known example.

Why it is fast

Speed in memory comes from two things: how fast each wire runs and how many wires there are. HBM goes for width.

A single HBM3E stack talks to the processor over a 1,024-bit interface. A standard DDR5 module uses 64 bits, and a graphics memory chip typically uses 32. HBM4, the next generation, doubles the stack interface to 2,048 bits. Because the wires are so short, each one can run at high speed without using much power.

The result is that one HBM3E stack can move more than a terabyte of data per second, and an accelerator carries several stacks.

Why AI needs it

A large language model is mostly a huge set of numbers, its weights. To produce each word of an answer, the chip has to read a large share of those weights out of memory. The arithmetic itself is quick; waiting for data is often the slow part.

That makes memory bandwidth, not raw processing power, the limit on how fast many AI workloads run. A processor with too little bandwidth sits idle. Memory capacity matters too: the more of a model that fits in HBM next to the chip, the fewer chips are needed to serve it. For how this differs from AI running on a phone, see on-device AI versus cloud AI.

Why it is scarce

It uses more silicon. An HBM die holds fewer bits for its size than a standard DRAM die, because space goes to the through-silicon vias and supporting circuits. Making the same number of bits as HBM takes several times the wafer area of ordinary memory.

Yields compound. A stack is only good if every die in it and every connection between them works. One faulty layer can ruin the whole tower, so taller stacks lose more units in production.

Packaging is a second bottleneck. Even with stacks in hand, they must be joined to the processor with advanced packaging, which only a few companies offer at scale and which takes years to expand.

Only three companies make it. SK Hynix, Samsung and Micron produce essentially all of the world's HBM. New memory factories cost billions of dollars and take years to build, and customers often reserve output well in advance under long contracts.

Why it matters

HBM decides how many AI accelerators can be built and how well they perform, so its supply shapes the pace of AI data center growth. It also reaches beyond data centers: every wafer a memory maker devotes to HBM is a wafer not making the DRAM used in PCs, phones and servers, which is why AI demand can push up memory prices for everyone else.

AI chips and data centers Nvidia

Spotted an error? Tell us and we will fix it.