Tensor cores are no longer enough: why AI accelerators will hit a memory wall

AI accelerator manufacturers are aggressively scaling computing power by increasing tensor core counts and adopting FP8 and FP4 data formats. However, this rapid performance growth is hitting a critical bottleneck: memory bandwidth. Modern chips often sit idle while waiting for model weights and intermediate data. To address this, Samsung is developing the HBM5 memory standard. This new technology promises to double performance compared to HBM4E while improving energy efficiency by 20%. With a 4096-bit bus, the bandwidth of a single stack could reach 4 TB/s. This development underscores that the future of AI computing depends not only on processing units but also on the memory's ability to supply them with the necessary data in real-time.