An Nvidia Rubin GPU carries up to 288GB of HBM4 memory and can move data to and from that memory at up to 22 terabytes per second. Those two numbers, not the chip’s 50 petaflops of low-precision math, are the ones that decide how fast a large language model answers you.
This is counterintuitive if you have spent years watching processors marketed on raw arithmetic throughput. But the arithmetic in a modern AI accelerator is largely idle, waiting. The reason is a gap that has been widening for twenty years: computing power has grown far faster than the ability to feed it.
The measurement most often cited comes from a Berkeley analysis of AI scaling, “AI and Memory Wall”, which found that peak server hardware FLOPS grew roughly 3x every two years while DRAM bandwidth grew about 1.6x and interconnect bandwidth about 1.4x over the same period. Compute has been outrunning memory by roughly a factor of two per cycle, compounding for two decades. That compounding gap is the memory wall.
Why AI is a bandwidth problem, not a math problem
Consider what happens when a model generates a single token of text. For each token, the accelerator must read the model’s weights out of memory, multiply them against a relatively small amount of activation data, and write results back. In a batch of one — a single user, a single chat — almost every weight in the model gets read from memory to produce one token.
The ratio of arithmetic operations to bytes moved is called arithmetic intensity, and for token-by-token generation it is terrible. The multiply-accumulate hardware finishes long before the next block of weights arrives. The accelerator sits idle, and utilisation of that expensive silicon can fall into the single digits.
Training behaves differently. Large batches and large matrix multiplications give you high arithmetic intensity, so training is far more compute-bound. This is why the industry’s centre of gravity matters: as spending shifts from training toward serving inference at scale, memory bandwidth becomes the dominant constraint on cost per token.
Capacity matters separately from bandwidth. A model whose weights and key-value cache do not fit in on-package memory must be split across several accelerators, and then you pay interconnect costs on every layer. Capacity determines whether the workload fits; bandwidth determines how fast it runs.
How HBM actually works
Conventional DRAM sits in modules alongside the processor and communicates over a relatively narrow bus running at high frequency. High-bandwidth memory takes the opposite approach: an extremely wide, comparatively slow bus placed extremely close to the processor.
An HBM stack is a tower of DRAM dies sitting on a base logic die. The dies are connected vertically by through-silicon vias — microscopic holes etched straight through the silicon and filled with conductor, so that signals travel a fraction of a millimetre vertically instead of centimetres across a motherboard. SK hynix, which pioneered much of this, describes TSVs as vertical electrodes linking the stacked chips through fine holes.
The finished stack sits on a silicon interposer next to the GPU die, inside the same package. That is what lets an HBM interface be thousands of bits wide: you cannot route 2,048 traces across a circuit board, but you can across a few millimetres of silicon.
The two hard parts
Stacking is not easy. Twelve or sixteen DRAM dies must be thinned dramatically so the finished stack still fits the package height budget. SK hynix’s 12-layer HBM3E made each chip about 40% thinner than in its 8-layer product to reach 50% more capacity at the same total height.
Heat is the other problem. A stack of memory dies is a poor thermal conductor with a hot GPU beside it, and the die at the bottom of the stack has the worst path out. Manufacturers use specialised moulding and bonding processes — SK hynix’s Advanced MR-MUF, and hybrid bonding at higher stack counts — largely to manage warpage and heat dissipation.
HBM3E and HBM4: the verified specifications
JEDEC, the standards body that defines memory specifications, published the base HBM3 standard in January 2022 with data rates up to 6.4 Gb/s per pin and 819 GB/s per device. HBM3E was an extension of that generation, keeping the 1,024-bit interface and pushing per-pin speed higher.
In production, SK hynix began volume production of 12-layer HBM3E in September 2024 at 36GB per stack and 9.6 Gbps per pin — which across a 1,024-bit interface works out to roughly 1.2 TB/s per stack.
HBM4 is the bigger change. JEDEC released the standard in April 2025. Its headline changes: the interface doubles to 2,048 bits, independent channels double from 16 to 32, and the specification allows up to 8 Gb/s per pin for a total of 2 TB/s per stack, with 4-high through 16-high configurations, 24Gb or 32Gb dies, and a maximum of 64GB per stack.
Both major suppliers are shipping above the JEDEC baseline. SK hynix announced completed HBM4 development in September 2025 at over 10 Gbps per pin across 2,048 I/O, claiming more than 40% better power efficiency than the prior generation. Micron entered high-volume production in March 2026 with 36GB 12-high stacks running above 11 Gb/s for more than 2.8 TB/s per stack, plus over 20% better power efficiency than HBM3E, and has sampled a 48GB 16-high part.
| HBM3 (JEDEC base) | HBM3E (shipping) | HBM4 (JEDEC) | HBM4 (shipping) | |
|---|---|---|---|---|
| Interface width | 1,024-bit | 1,024-bit | 2,048-bit | 2,048-bit |
| Per-pin speed | 6.4 Gb/s | 9.6 Gbps | Up to 8 Gb/s | 10–11+ Gb/s |
| Bandwidth per stack | 819 GB/s | ~1.2 TB/s | Up to 2 TB/s | Over 2.8 TB/s |
| Max capacity per stack | 64GB (32Gb 16-high) | 36GB (12-high shipping) | 64GB (32Gb 16-high) | 36GB 12-high; 48GB 16-high sampling |
| Independent channels | 16 | 16 | 32 | 32 |
Note the odd detail in that table: shipping products exceed the standard’s headline speed. JEDEC specifies a floor for interoperability; vendors compete above it. When you see conflicting HBM4 bandwidth figures, this is usually why.
The three-way race, and how it has shifted
Only three companies make HBM at scale, and the ranking has moved. Counterpoint Research put Q2 2026 HBM market share at SK hynix 50%, Samsung 33% and Micron 18%.
SK hynix built its lead by being first and being reliable, winning the bulk of Nvidia’s business through the HBM3 and HBM3E generations. Samsung, the largest DRAM maker overall, spent much of 2024 and 2025 struggling with HBM qualification and has clawed back share since. Micron is the smallest of the three but has moved quickly, reaching high-volume HBM4 production in early 2026.
The competitive picture is genuinely fluid, and share numbers from different analyst firms disagree by several points depending on how they count revenue versus bits. Treat any single figure as a snapshot.
Why HBM is making your laptop more expensive
This is where the memory wall stops being an engineering story and becomes an economic one. HBM is DRAM, made on the same fabs, competing for the same wafers as the memory in phones, laptops and servers. IDC’s analysis of the 2026 shortage put it bluntly: every wafer allocated to an HBM stack is a wafer denied to a smartphone memory module.
HBM also consumes more wafer area per gigabyte than commodity DRAM, because of the logic die, the TSV overhead and the yield loss inherent in stacking twelve or sixteen dies without a single failure. And it carries much higher margins, so manufacturers have every reason to prioritise it.
IDC projected 2026 DRAM supply growth at 16% year-on-year, below historical norms, and modelled potential smartphone market contraction of 2.9% to 5.2% and PC market decline of 4.9% to 8.9% as a result of memory scarcity and rising prices. Those are forecast scenarios, not measured outcomes, but the direction has been visible in retail DDR5 and SSD pricing throughout 2026.
CXL and the alternatives
Compute Express Link is the most discussed alternative, and it is important to be precise about what it does. CXL is a cache-coherent interconnect layered on top of PCIe. It lets you attach large pools of ordinary DRAM to a server, share memory between hosts, and allocate capacity flexibly instead of stranding it in individual machines.
What CXL does not do is solve the bandwidth problem. Attached memory runs at PCIe speeds with added latency — useful for capacity expansion, memory pooling and reducing waste in general-purpose data centres, but nowhere near HBM’s in-package bandwidth. CXL and HBM address different halves of the problem.
Other approaches under active work:
- Larger on-die SRAM. Some AI chip startups put the entire model in on-chip SRAM, which has enormous bandwidth but tiny capacity and terrible cost per gigabyte.
- Processing in memory. Moving simple arithmetic into the memory die itself, so data does not have to travel. Promising in research, still niche commercially.
- Quantisation and sparsity. The most effective lever available today. Running a model in 4-bit instead of 16-bit cuts the bytes you must read per token by four, which is a direct multiplier on inference speed.
- Better batching and caching. Serving many requests together raises arithmetic intensity. Most of the practical throughput gains in production inference over the last two years came from software of this kind, not from hardware.
What this means for you
If you are running or buying AI inference, compare accelerators on memory capacity and bandwidth first. A chip with more peak FLOPS and less bandwidth will usually lose on tokens per second per dollar for generative workloads. Ask what the model plus its key-value cache actually requires, then find the cheapest hardware that fits it with bandwidth to spare.
If you are self-hosting models, quantisation is the highest-leverage change you can make. Dropping from 16-bit to 8-bit or 4-bit weights reduces both the capacity you need and the bytes read per token, which improves latency roughly proportionally. Quality loss on well-implemented modern quantisation schemes is often smaller than people expect, though it is real and worth measuring on your own workload.
If you are buying a laptop, phone or building a PC, expect memory to be a larger share of the bill of materials than it was in 2024, and expect that to persist while AI capacity expansion continues. Buying more RAM up front on a machine with soldered memory has become a better-value decision than it used to be.
If you are budgeting for AI at a company level, understand that the cost of serving a model is set largely by memory economics. This is also why context length is expensive: a longer context means a larger key-value cache, which consumes both capacity and bandwidth on every single token generated.
Frequently asked questions
Why not just add more HBM stacks to a GPU?
Physical space on the interposer and package limits how many stacks fit around a die, and each stack adds power and heat. Nvidia’s Rubin reaches up to 288GB per GPU and 22 TB/s aggregate, and going further generally means bigger packages, more power, and lower yields.
Is HBM the same as the GDDR in a gaming graphics card?
No. GDDR is high-speed memory on a narrow bus soldered near the GPU on a board. HBM is a wide, slower-clocked, stacked memory inside the same package. GDDR is far cheaper per gigabyte; HBM offers several times the bandwidth. Consumer graphics cards use GDDR for cost reasons.
Will HBM4 fix the memory wall?
It narrows the gap for one generation. Roughly doubling per-stack bandwidth is a large step, but compute is still scaling faster. Unless that trend reverses, each memory generation buys time rather than solving the problem.
Does the memory wall affect model training too?
Less severely. Training runs large batches with high arithmetic intensity, so it is more compute-bound. Training is instead constrained by memory capacity and by interconnect bandwidth between accelerators when a model is split across many chips.
Are DRAM prices going to come back down?
Eventually, if capacity expansion outruns demand — memory has always been cyclical. But new fab capacity takes years to come online, and manufacturers are cautious about repeating past overbuilds. Nobody credibly forecasts a rapid correction.
The constraint that is shaping the industry
For most of computing history, memory was a supporting component. It is now the thing that decides what an AI system costs to run, which chip wins a design, and, increasingly, what you pay for a phone.
HBM4 hardware is shipping in volume as of 2026 and the specification already contemplates 16-high stacks and 64GB per stack. But the underlying arithmetic has not changed: compute keeps scaling faster than the pipes that feed it. The most durable wins are likely to come from making models move fewer bytes — quantisation, sparsity, smarter caching, architectures with lower memory traffic — rather than from waiting for the next memory generation to rescue the design.
Sources
- arXiv — AI and Memory Wall (Gholami et al.)
- JEDEC — Release of the JESD270-4 HBM4 standard
- JEDEC — HBM3 update to the High Bandwidth Memory standard
- SK hynix Newsroom — Volume production of the world’s first 12-layer HBM3E
- SK hynix Newsroom — SK hynix completes world’s first HBM4 development
- Micron — High-volume production of HBM4 designed for NVIDIA Vera Rubin
- Tom’s Hardware — Micron enters high-volume production of HBM4
- NVIDIA Technical Blog — Inside the NVIDIA Vera Rubin platform
- Counterpoint Research — Global DRAM and HBM market share
- IDC — Global memory shortage crisis and its impact on smartphones and PCs in 2026
- CXL Consortium — About Compute Express Link
- Tom’s Hardware — JEDEC finalizes HBM4 memory standard
Image credit: Photo: L.Willms — CC BY-SA 3.0 (via Wikimedia Commons)
