The AI memory bottleneck.

For two years the public conversation about AI infrastructure has fixated on compute — TFLOPs, GPU counts, training runs measured in megawatts. That framing is increasingly wrong. The frontier-model bottleneck is now memory bandwidth: how quickly model weights and activations can be moved from memory to the compute cores that need them. By every internal benchmark from the major labs, modern transformer inference is memory-bound 70–85% of the time, and training is bandwidth-limited in roughly the same proportion at the scales now common.

The math is unforgiving. A 70B-parameter model in FP16 occupies ~140 GB. To generate a single token, the entire weight set must be read from memory and processed. An H200 with 4.8 TB/s of HBM3E bandwidth can theoretically push ~34 reads/second of that model — which is your ceiling for tokens-per-second per GPU before any other factor enters. Compute is rarely the wall. Bandwidth is.

We used to design accelerators around peak FLOPs. We now design them around bytes per FLOP — and the answer is always "more, faster, closer."— Sandra Mehta, lead architect, anonymous hyperscaler · April 2026

This reframing matters because the supply chain for bandwidth looks nothing like the supply chain for compute. There is functionally one company on Earth — TSMC — that can do advanced packaging at the volumes needed. There are three companies that can manufacture leading-edge HBM stacks. There is a multi-quarter queue for both. Read the long-form deep-dive →

HBM is what you get when you stack 8 to 16 DRAM dies vertically, drill thousands of through-silicon vias (TSVs) through them, and bond the stack directly to a GPU's silicon interposer. The result is an order-of-magnitude bandwidth uplift versus DDR — H200 HBM3E delivers 4.8 TB/s; the fastest server DDR5 manages ~70 GB/s.

But HBM stacks are catastrophically harder to produce than commodity DRAM. Yields fall with each additional die layer; thermal management between layers is a research-grade problem; TSV pitch is now under 40µm and shrinking; and every stack must be co-packaged with the GPU via TSMC's CoWoS or equivalent. The packaging line, not the memory fab, is the binding constraint today.

Three implications follow from this gap. First, HBM allocation is now a strategic asset: NVIDIA, AMD, and the hyperscalers negotiate multi-year supply agreements that are functionally equivalent to capacity ownership. Second, CoWoS capacity at TSMC is the harder constraint than HBM itself through 2026. Third, alternative paths — see the next section — are no longer fringe research. They are board-level priorities at every relevant company.