OpenAI’s Jalapeno AI Chip Could Challenge NVIDIA’s CUDA Advantage With a Powerful Custom ASIC
OpenAI appears to be moving deeper into custom silicon with a new AI accelerator known as Jalapeno, a chip reportedly designed with help from its current AI models running on NVIDIA GPUs. The twist is hard to miss: the same GPU ecosystem that helped fuel OpenAI’s rise may now be helping it build hardware that could eventually reduce its dependence on NVIDIA.
Jalapeno is not just another AI chip. It is described as a large-scale ASIC built specifically for artificial intelligence workloads, especially inference, where trained models generate answers, predict tokens, and serve users at massive scale. Paired with OpenAI’s own Gluon kernel programming language, Jalapeno could become part of a broader attempt to chip away at NVIDIA’s long-standing CUDA software advantage.
At the center of Jalapeno is a single reticle-sized compute die manufactured on TSMC’s N3P process. This compute die is paired with an N3E I/O chiplet and HBM4 memory, reportedly delivering 15.4 TB/s of memory bandwidth per package. That level of bandwidth is critical for modern AI, where performance is often limited not by raw compute alone, but by how quickly data can move between memory and processing units.
The chip’s architecture divides its cores and memory into matching slices. Each compute slice gets a low-latency local view of its assigned HBM slice, while a high-bandwidth collective network connects the system together. This design is meant to keep data close to where it is needed, reducing delays and improving efficiency during large AI workloads.
To understand why this matters, it helps to look at how AI models actually work. Large language models do not process information like a typical program reading one line at a time. They handle enormous grids of numbers called matrices. These matrices represent prompts, probabilities, model weights, and the mathematical relationships that allow AI systems to generate useful responses.
That is where Jalapeno’s matrix engines come in. These engines are the heavy lifters of the chip, built to accelerate the repetitive matrix math at the heart of AI inference.
Instead of relying only on traditional 16-bit or 8-bit number formats, Jalapeno uses MXFP numerical formats. These formats compress AI data even further, with MXFP4 reducing numbers down to 4-bit precision. The key idea is that groups of numbers can share a scaling factor, allowing the system to store and move less data while still preserving enough accuracy for many AI operations.
This is important because memory movement is one of the biggest bottlenecks in AI computing. If a chip spends too much time pulling data from memory and writing results back, raw processing power goes underused. Jalapeno tries to solve that problem with a weight-stationary systolic array.
A systolic array works like a tightly connected grid of processing cells. Data flows through the grid from one cell to the next, instead of constantly traveling back and forth to external memory. In a weight-stationary design, the model’s weights are loaded into the processing grid and kept in place while incoming data flows through them.
For AI inference, this is a major advantage. The model weights are often massive and reused repeatedly. By keeping them stationary, Jalapeno cuts down on unnecessary data movement. Combined with MXFP compression and systolic data flow, the chip is designed to remove many of the bottlenecks that slow down large AI models.
Jalapeno also includes 64-bit scalar cores that handle control code, memory management, and overall coordination. These scalar cores use out-of-order execution pipelines and are paired with L1 cache, allowing them to keep working instead of stalling while waiting for data.
In practical terms, these scalar cores help prepare work for the matrix and vector engines. They stream data into specialized hardware queues so that when the high-throughput compute units are ready, the numbers are already lined up. This helps the chip maintain high utilization during demanding inference workloads.
The chip also includes FP32 and INT32 vector cores. These are useful when higher precision is needed and compressed matrix math would not be ideal. Tasks such as softmax calculations, final probability generation, and layer normalization can require more precise computation, and Jalapeno’s vector cores are designed to handle those cases.
OpenAI’s plans reportedly go beyond a single chip. Jalapeno is said to be part of a complete rack-scale AI system built around two major rack types.
The CPU host rack, referred to as Katsu, contains 16 host CPU trays. Each tray reportedly includes two Turin-class AMD EPYC processors, 1.5TB of standard DRAM, local NVMe SSD storage, and 400G frontend networking.
The ASIC accelerator rack, referred to as Vindaloo, sits alongside the host rack and contains 16 accelerator trays. Each tray houses 8 Jalapeno ASICs, bringing the total to 128 Jalapeno chips per rack deployment.
The two racks are connected using high-speed PCIe Direct Attach Copper cables, with each host tray linked directly to its matching accelerator tray. This layout is designed to give the system a clean and direct path between CPU resources and AI acceleration hardware.
A full 128-chip Jalapeno rack is said to deliver up to 1.7 ExaFLOPs of 4-bit MXFP4 processing power. It also includes 27.5TB of HBM4 memory per rack. Each Jalapeno package reportedly uses six 12-high HBM4 stacks, enabling the 15.4 TB/s memory bandwidth figure per package.
Performance claims around Jalapeno are especially notable. In reported inference benchmark results, the chip achieved 1.5x to 1.9x more AI work per watt at peak throughput compared with the strongest NVIDIA GB200 and GB300 results available during testing. It also showed 1.7x to 3.6x lower end-to-end latency in those comparisons.
That matters because inference efficiency is becoming one of the most important metrics in AI. Training large models gets plenty of attention, but inference is where companies spend heavily every day to serve real users. Lower latency means faster responses. Better performance per watt means lower operating costs. At massive scale, those savings can be enormous.
Jalapeno is also said to be flexible rather than tuned for just one model family. If accurate, that would make it more useful for a wide range of AI systems instead of being limited to a narrow set of workloads.
The chip reportedly delivers 13.4 PFLOPs of MXFP4 performance on its reticle-sized compute die using TSMC’s N3P process. For comparison, NVIDIA’s upcoming Rubin-class compute die has been described as offering 17.5 PFLOPs of dense low-precision performance on a similarly sized die and process node. While Jalapeno may not lead in every raw compute metric, its efficiency and system-level design appear to be its main strengths.
Power consumption is another important part of the story. Jalapeno is reportedly rated at a 700W TDP, but during active AI workloads its sustained power use is said to remain at or below 550W. If that holds true in large-scale deployment, it could give OpenAI a meaningful advantage in data center efficiency.
One of the most interesting details is that Jalapeno reportedly achieves its results using a Single-Token Prediction architecture. In simple terms, the model processes a prompt and predicts one token at a time, which is the traditional way large language models generate text.
That is important because some performance gains in AI inference can come from techniques such as speculative decoding or multi-token prediction, which can improve throughput but complicate comparisons. Jalapeno’s reported numbers do not appear to rely on those methods to boost results. Speculative decoding alone can improve cost per token significantly, so the fact that Jalapeno performs well without leaning on it makes the architecture more compelling.
There is one important caveat. Jalapeno reportedly uses HBM4 memory, while NVIDIA GB200 and GB300 systems use HBM3E. HBM4 offers higher bandwidth, which gives Jalapeno a natural advantage in some comparisons. That does not erase the chip’s apparent strengths, but it does mean direct benchmark comparisons should be viewed with context.
Even with that caveat, Jalapeno could represent a major moment in the AI hardware race. OpenAI has built some of the world’s most demanding AI services, and that gives it a powerful incentive to design chips around its own workloads. If Jalapeno performs as described, it may help OpenAI reduce costs, improve latency, and gain more control over the hardware and software stack behind its AI models.
The bigger story is not just that OpenAI may have a fast AI accelerator. It is that the company could be building an alternative ecosystem around custom silicon and its own programming tools. NVIDIA’s CUDA platform has long been one of the strongest barriers protecting its dominance in AI computing. If OpenAI can combine Jalapeno hardware with a capable software layer, it could begin to challenge that advantage from the inside out.
For now, NVIDIA remains the dominant force in AI infrastructure. But Jalapeno suggests that the next phase of the AI race may not only be about bigger models. It may also be about who controls the chips, memory, software, and data centers that make those models possible.
