Groq Reportedly Turns to 3D Stacked DRAM for Next-Generation AI Chips
Groq is reportedly preparing a major memory shift for its next generation of AI chips, moving beyond its current SRAM-heavy design and exploring 3D stacked DRAM technology. The change could allow future Groq processors to handle much larger AI workloads by prioritizing memory capacity while still aiming to preserve strong bandwidth performance.
The company is already known for its high-speed AI inference hardware. Its current Groq LPU architecture uses 500 MB of SRAM per chip, delivering extremely high bandwidth of up to 150 TB/s. In rack-scale configurations, 256 LPUs can be combined to provide around 128 GB of SRAM capacity and roughly 640 TB/s of scale-up bandwidth. Each LPU is also said to deliver up to 1.20 PFLOPs of FP8 compute performance, making the platform especially focused on fast AI inference.
Earlier this year, Groq LPX racks gained attention as part of a broader push to improve AI inference performance in large-scale infrastructure. These racks entered full production in August, and early performance figures showed up to 3,400 tokens per second when paired with NVL72 server configurations. That level of throughput highlights why Groq’s architecture has become a point of interest in the AI hardware market, especially as demand grows for faster and more efficient inference systems.
Now, the company appears to be preparing for a different approach. Instead of relying only on SRAM, Groq is reportedly planning to use DRAM in a 3D stacked design for future chips. This would represent a significant architectural shift.
SRAM is extremely fast and offers excellent bandwidth, but it has a major limitation: capacity. DRAM, on the other hand, is not as fast as SRAM, but it can provide much higher memory density. For modern AI workloads, especially large language models and increasingly complex generative AI systems, capacity is becoming just as important as speed. Larger models require more memory to store parameters, context, and intermediate data, and that is where DRAM can offer a major advantage.
According to industry chatter, Groq’s next-generation design may use 3D packaging to stack DRAM layers vertically. This approach is similar in concept to how high-bandwidth memory is built today. By stacking memory layers on top of each other and connecting them with through-silicon vias, or TSVs, chipmakers can increase capacity without expanding the chip’s footprint as much as traditional memory layouts would require.
The benefits of 3D stacked DRAM could be especially important for AI accelerators. A vertically stacked memory design can reduce surface area, improve power efficiency, and enable more memory to sit closer to the compute engine. While DRAM generally cannot match SRAM’s raw bandwidth, advanced packaging techniques may help narrow the gap while giving Groq a major boost in memory capacity.
This suggests Groq may be preparing its hardware for the next phase of AI computing, where model size and memory efficiency are becoming critical. AI inference is no longer just about raw compute. As models become larger and more widely deployed, hardware platforms need to balance speed, memory capacity, power consumption, and scalability.
Groq’s reported move does not necessarily mean the company is abandoning SRAM entirely. Instead, it may continue offering separate SRAM-based and DRAM-based solutions depending on workload needs. SRAM could remain ideal for ultra-low-latency, high-bandwidth tasks, while 3D stacked DRAM may be better suited for larger models that need more memory capacity.
That flexibility could make Groq’s future AI chips more attractive to data centers, cloud providers, and companies building large-scale AI platforms. If the company can combine its inference-focused architecture with higher memory capacity, it could strengthen its position in the rapidly expanding AI accelerator market.
The shift also reflects a broader trend across the semiconductor industry. As AI workloads grow, chip designers are increasingly turning to advanced packaging and stacked memory technologies to overcome the limits of traditional chip design. More compute power alone is not enough. The memory system has become one of the most important parts of AI hardware performance.
For Groq, 3D stacked DRAM could be the next step in scaling its LPU technology for larger and more demanding AI models. While SRAM helped the company deliver impressive bandwidth and inference speed, DRAM may give its next-generation chips the capacity needed for the future of artificial intelligence.
If these plans move forward, Groq’s upcoming AI chips could mark an important evolution in the company’s strategy: less focus on memory speed alone, and more emphasis on delivering the right balance of capacity, efficiency, and performance for next-generation AI inference.
