d-Matrix Says 3D DRAM Is Ready to Tackle AI’s Growing Memory Bottleneck
AI hardware is entering a new phase where raw compute power is no longer the only challenge. As large language models grow and KV cache demands expand, memory bandwidth, capacity, power consumption, and data movement are becoming some of the biggest limits in modern AI systems. d-Matrix believes the answer is not simply more HBM or larger SRAM pools, but a new approach: 3D DRAM.
The company is proposing its Raptor 3D DRAM technology as a practical solution for AI inference workloads that need massive bandwidth without the extreme power costs associated with traditional high-bandwidth memory. According to d-Matrix, 3D DRAM can deliver SRAM-class bandwidth while offering far better capacity than SRAM and much higher energy efficiency than HBM.
Today’s AI accelerators generally rely on two major memory options: SRAM and HBM. SRAM is extremely fast and efficient, making it ideal for high-bandwidth workloads. However, it is difficult to scale in capacity, suffers from leakage, and is far more expensive than HBM. That makes SRAM suitable for smaller models or specific accelerator designs, but not ideal as a broad solution for large-scale AI deployments.
HBM, on the other hand, offers far greater capacity and is widely used in high-end AI GPUs and accelerators. The problem is that scaling HBM bandwidth is becoming increasingly difficult. Power limits, packaging complexity, I/O constraints, and physical space around the chip all create barriers. Current HBM4-class solutions can deliver around 20 TB/s of bandwidth, but future AI inference workloads may require far more.
d-Matrix argues that pushing HBM to 100 TB/s would come with a major energy penalty. At roughly 2.4 pJ per bit, a 100 TB/s HBM solution could require close to 2 kW of power for the memory alone. Once data movement is considered, the total system power burden becomes even more significant. For AI data centers already dealing with rising energy demands, this is a serious problem.
This is where 3D DRAM enters the picture. Instead of placing memory next to the compute logic, 3D DRAM stacks custom DRAM and logic vertically using short, dense interconnects. The goal is to reduce data movement, increase bandwidth, improve efficiency, and avoid some of the physical limitations that come with conventional HBM designs.
d-Matrix’s Raptor 3D DRAM uses a logic die built on TSMC’s 4 nm process stacked face-to-face on top of custom DRAM using a 36-micron pitch. This design allows the compute logic and memory to communicate through very short vertical connections, reducing the energy required to move data and enabling extremely high bandwidth.
The company claims Raptor can deliver more than 100 TB/s of bandwidth with 32 GB of capacity in a single-high stack. Its measured energy efficiency is listed at 0.37 pJ per bit, which is dramatically lower than HBM-class memory. Compared with a 192 GB HBM4 solution delivering around 18 TB/s at roughly 2 to 3 pJ per bit, d-Matrix says Raptor offers about 5.6 times higher bandwidth, 5 to 8 times lower energy, and significantly denser I/O.
The design is not meant to fully replace HBM in every scenario. HBM still offers higher total capacity, which remains important for large training systems and memory-heavy applications. Instead, d-Matrix is positioning 3D DRAM as a middle ground between SRAM and HBM. It provides much more capacity than SRAM, far more bandwidth than HBM, and much better energy efficiency than conventional off-chip memory.
This balance is particularly important for AI inference, especially the decode phase of large language model workloads. In many LLM and agentic AI applications, performance is limited less by compute and more by how quickly data can be accessed and moved. A memory architecture that delivers extremely high bandwidth at lower power could significantly improve throughput and efficiency.
One of the central ideas behind Raptor is aligning the memory banks closely with the compute engines. The chiplet contains 256 tensor engines, and d-Matrix pitch-matches the DRAM banks to these engines to reduce unnecessary data movement. Instead of relying on a few large systolic arrays with data traveling longer distances across the chip, the design places memory access closer to where computation happens.
However, 3D DRAM is not without challenges. One issue is bank-to-channel mapping. The DRAM structure does not naturally align with the 128-byte flits expected by the tensor engines. Each channel provides three banks, producing 96 bytes, while the compute fabric expects 128 bytes. Raptor can compensate through overfetching and aggregation, but the design must be carefully optimized to avoid wasting bandwidth.
Another challenge is I/O power. Even at 0.37 pJ per bit, delivering 100 TB/s of bandwidth still means roughly 300 W of I/O power. That is far lower than a comparable HBM approach, but still a meaningful amount of heat and energy that must be managed within a dense accelerator package.
Thermals are also a major concern for 3D-stacked memory. d-Matrix has chosen a logic-on-top design, which makes it easier to attach cooling directly to the logic die using a cold plate. The company says a single-high stack with power density below 0.5 W/mm² can be reliably liquid-cooled while keeping the DRAM under safe temperature limits.
Because the DRAM may operate near a 105°C junction temperature, refresh behavior becomes important. As temperature rises, DRAM leakage increases, requiring more frequent refresh cycles. Raptor refreshes every 4 milliseconds, around eight times faster than typical DRAM refresh behavior. Despite this, d-Matrix says the bandwidth loss from refresh is only about 1.37%, which is relatively small given the overall performance target.
Reliability and yield are also addressed through the architecture. Raptor uses small microbanks, each with 1,366 rows and 5.33 MB of capacity. The design includes around 8 to 9% spare banks and uses a redundancy system to preserve channel symmetry. For error correction, the logic die includes Reed-Solomon T=2 error correction, capable of correcting two symbol errors across 128 bytes, along with CRC protection.
These reliability features are important because stacked memory architectures introduce new manufacturing and operational risks. By designing redundancy and strong error correction directly into the system, d-Matrix aims to make 3D DRAM practical for high-volume AI accelerator deployments rather than just a laboratory concept.
In performance comparisons, d-Matrix positions Raptor 3D DRAM as a major improvement over HBM4-based platforms. With effective bandwidth utilization around 83 to 85%, the company claims Raptor can deliver more than 23 times higher bandwidth density in GB/s per mm² and around 13.5 times better power efficiency in mW per GB/s compared with an HBM4-based AI platform.
The broader message is clear: AI workloads are becoming increasingly memory-bound, and traditional memory scaling may not be enough. As model sizes, context lengths, and inference demand continue to grow, accelerators need memory systems that can deliver more bandwidth without consuming unsustainable amounts of power.
3D DRAM could provide that path. By stacking logic and DRAM together with dense vertical interconnects, it reduces the distance data must travel, improves energy efficiency, and provides bandwidth levels closer to SRAM while maintaining useful capacity. It also avoids some of the packaging and I/O limits that make HBM difficult to scale beyond current generations.
For AI data centers, the potential benefits are significant. More bandwidth per watt can mean faster inference, higher token throughput, lower operating costs, and better performance density. As companies race to deploy more efficient AI infrastructure, memory architecture may become just as important as compute architecture.
d-Matrix’s Raptor 3D DRAM is not just another memory specification. It represents a shift in how AI accelerators may be designed in the future, with memory and compute becoming more tightly integrated to overcome the limitations of traditional chip packaging. If the technology scales as promised, 3D DRAM could become a key building block for next-generation AI inference systems.
The question now is how quickly the rest of the memory and AI hardware industry will respond. HBM will continue to evolve, and SRAM will remain important for specialized workloads, but the growing pressure from AI data movement may create a strong opening for 3D DRAM. For d-Matrix, the timing is clear: the era of 3D DRAM for AI acceleration may have already begun.
