High-Bandwidth Flash could ease the AI memory bottleneck by taking pressure off HBM
The race to build larger, faster AI systems is running into one of the industry’s biggest obstacles: memory. Modern AI accelerators are no longer limited only by raw compute power. Increasingly, they are constrained by how much high-speed memory they can access, how quickly they can move data, and how expensive it is to scale that memory.
That is why SanDisk’s work on High-Bandwidth Flash, or HBF, in collaboration with SK hynix, is drawing attention. The technology could create a new memory tier for AI hardware, allowing some workloads currently handled by High-Bandwidth Memory, or HBM, to move onto a denser and potentially more cost-effective flash-based solution.
The idea is not to replace HBM entirely. Instead, HBF could take over specific read-heavy tasks, such as storing AI model weights, while HBM remains responsible for workloads that require frequent and fast writes, such as KV cache. If successful, this split could help reduce one of the biggest bottlenecks in AI infrastructure.
Why HBM is under pressure in AI systems
To understand why HBF matters, it helps to look at the role of HBM in today’s AI hardware.
Large language models need vast amounts of memory simply to store their parameters. For example, an AI model with 100 billion parameters running at fp16 precision uses 2 bytes per parameter. That means the model needs around 200GB of memory just to hold its weights.
Weights are the stored knowledge of an AI model. They determine how much importance the model gives to specific concepts, patterns, words, or relationships. In general, the larger and more capable the model, the more weights it needs, and the more memory it consumes.
But model weights are only part of the problem.
AI models also rely on something called the Key-Value cache, commonly known as KV cache. This is especially important during inference, when a model generates text, code, images, or other outputs.
A simple way to understand KV cache is to imagine writing a long story while having poor short-term memory. Every time you add a new word, you would need to reread everything you have already written to remember the context. That would be painfully slow. KV cache acts like a set of notes that helps the model remember what has already been processed, allowing it to generate responses much faster.
As the context window grows, the KV cache also grows. Longer prompts, longer conversations, and larger documents all require more cache. Because the model needs to access and update this information quickly, KV cache is usually stored in HBM.
This creates a major challenge. HBM has exceptional bandwidth, but it is expensive, physically limited, and tightly integrated near the GPU or AI accelerator. A typical 12-Hi HBM4 stack may offer around 36GB of capacity, while a 16-Hi version may reach about 48GB. Scaling capacity often means adding more accelerators, which dramatically increases system cost.
In other words, HBM is being asked to do two demanding jobs at once: store massive model weights and handle rapidly changing KV cache. That is where High-Bandwidth Flash could change the equation.
How High-Bandwidth Flash works
High-Bandwidth Flash is designed to bring NAND flash into a high-bandwidth memory-like package. Just as HBM stacks DRAM dies vertically, HBF stacks NAND dies on top of one another. These dies are connected using Through-Silicon Vias, or TSVs, with a controller logic die bonded to the stack.
The proposed HBF design is expected to offer up to 512GB of storage with bandwidth ranging from around 0.4TB/s to 3TB/s. That combination of high capacity and strong read bandwidth could make it highly attractive for AI workloads that need to repeatedly read large volumes of data.
The key difference is that HBF uses NAND flash rather than DRAM. NAND is far denser and can store more data in a smaller space, but it also has a major drawback: it is much slower at the individual cell level.
For comparison, SRAM can deliver read access in roughly a nanosecond. DRAM is slower, at around 100 nanoseconds. NAND flash is much slower still, often around 100 microseconds. On a per-cell basis, NAND can be about 1,000 times slower than DRAM.
At first glance, that sounds like a deal-breaker. However, HBF attempts to overcome this weakness through massive parallelism.
Instead of relying on a small number of fast reads, the HBF logic die can schedule thousands of NAND reads at the same time. Each individual read may be slow, but the combined output of many parallel reads can produce very high aggregate bandwidth. This is how HBF aims to deliver bandwidth in the range of hundreds of gigabytes per second to multiple terabytes per second.
Why HBF is better suited for model weights than KV cache
The most important limitation of HBF is write performance. NAND flash is not only slower when writing data, but it also has limited write endurance compared with DRAM. Repeated writes wear out NAND cells over time.
That makes HBF a poor fit for KV cache, which changes constantly as an AI model processes new tokens. KV cache requires frequent updates and very low-latency access, so HBM will remain the better choice for that job.
Model weights are different. Once loaded, they are mostly read rather than rewritten. During inference, the model repeatedly accesses its weights to generate outputs, but those weights do not need to be updated constantly. This makes them a much better match for HBF’s strengths: high capacity and high read bandwidth.
By moving model weights from HBM to HBF, AI accelerators could free up valuable HBM capacity for KV cache. That could allow systems to support longer context windows, larger prompts, and more complex workloads without requiring as much expensive HBM.
What this could mean for future AI hardware
If High-Bandwidth Flash reaches commercial deployment, it could help reshape AI memory architecture. Instead of relying almost entirely on HBM for both static and dynamic data, future systems could use a tiered approach.
HBF could store large, read-heavy AI model weights.
HBM could handle fast, frequently updated KV cache.
GPU or accelerator compute units could access both memory pools depending on workload needs.
This kind of memory separation could deliver several benefits. It could reduce the pressure to increase HBM capacity with every new AI hardware generation. It could also make larger AI models easier to deploy by providing a denser memory tier for weight storage. Most importantly, it could help improve cost efficiency in large-scale AI data centers.
However, HBF should not be confused with ordinary SSD storage. Although it uses NAND flash, it still requires advanced packaging, stacked dies, TSV connections, and dedicated logic. That means it will not be as cheap as standard solid-state storage. Even so, it may be meaningfully less expensive than HBM, especially at high capacities.
The bigger picture for AI memory
The AI industry is hungry for more memory bandwidth and more memory capacity. As models grow larger and context windows expand, the traditional approach of simply adding more HBM becomes increasingly expensive and difficult to scale.
High-Bandwidth Flash offers a possible middle path. It does not need to beat HBM at everything. It only needs to be good enough at the right task: storing and serving model weights with high read bandwidth.
If SanDisk and SK hynix can turn HBF into a practical standard, it could become a major new tool for AI infrastructure. HBM would still remain essential, especially for low-latency and write-heavy workloads. But HBF could help remove some of the burden, allowing AI systems to use HBM more efficiently.
That could lead to larger models, longer context windows, better inference performance, and more cost-effective AI deployments. In a market where memory has become one of the biggest constraints on progress, High-Bandwidth Flash may be one of the most important technologies to watch.
