OpenAI’s Jalapeño chip is built for fast inference at scale, benchmarks show

OpenAI pulled back the curtain on its custom silicon project, Jalapeño, during this week’s Hot Chips conference. The company provided a deeper technical dive into the architecture and, for the first time, shared performance data that suggests the chip is poised to challenge current industry leaders.

Benchmarking Against the Best

When subjected to the SemiAnalysis InferenceX benchmark, Jalapeño demonstrated superior efficiency compared to existing state-of-the-art inference processors. The results indicate that the chip delivers a higher volume of tokens per user while simultaneously achieving greater throughput per kilowatt.

“The bottom line is that the results show a very, very significant performance advance over state of the art,” said Richard Ho, OpenAI’s head of hardware. “Jalapeño can serve more AI work per unit of power, while also returning responses more quickly.”

While these figures were measured against Nvidia’s Blackwell systems, OpenAI acknowledges that the competitive landscape is fluid. By the time Jalapeño hits the market, Nvidia and other hardware giants will likely have introduced their own advancements. OpenAI currently projects a limited rollout by the end of 2026, with a broader deployment expected in 2027.

A Full-Stack Approach to Inference

First unveiled last October, Jalapeño is the product of a strategic partnership with Broadcom. Notably, OpenAI utilized its own internal AI models to assist in the design and development of the hardware. This "full-stack" philosophy allows the company to synchronize the evolution of its software models with the underlying silicon and memory architecture.

Key Architectural Advantages

OpenAI’s design strategy focuses on eliminating common friction points that typically plague inference processing. The company highlighted several technical priorities:

• Bottleneck Mitigation: The architecture is specifically tuned to reduce delays during the prefill and communication phases.
• Data Localization: By keeping the model state—including the KV cache—local, the system avoids the latency penalties associated with constant data movement.
• Optimized Resource Activation: The chip intelligently coordinates compute, memory, and networking resources to match the specific requirements of each inference stage.

By controlling the entire stack, OpenAI aims to create a multigenerational platform that remains efficient even as its models grow in complexity and demand.