OpenAI Jalapeño: The Architectural Decisions Behind Its Inference Efficiency

Jalapeño is OpenAI’s first custom-silicon inference chip, jointly developed with Broadcom and manufactured on TSMC's 3nm process. It was officially introduced two months before its presentation at the 2026 Hot Chips conference. At the time, limited information was available about the chip and its claims of breakthrough inference performance per Watt.

The benchmark results have attracted attention, and a closer look at the design choices provides important context for understanding the performance. This article presents TechInsights’ commentary on the decisions disclosed at Hot Chips.

Figure 1 – OpenAI Jalapeño headlines (Source: OpenAI HotChips 2026)

Locality First or Jalapeño's Innovative Memory and Core Architecture

Based on OpenAI’s patents, Jalapeño uses a weight-stationary systolic array, placing it within the same broad architectural family as Google’s TPU. In this design, the model’s learned parameters (weights) are loaded once and remain in place while input data streams through the array. This reduces power-consuming data movement, which is particularly relevant to inference because the same learned parameters are queried billions of times.

Jalapeño extends that focus on local data access through a slice-based memory architecture. Each core slice has a low-latency, local view of its own HBM slice, keeping the KV cache and weights localized. The resulting memory hierarchy is simpler and more predictable. Explicit memory slices handle local data access and are supported by a general network-on-chip that can overlap with compute.

That choice creates a different requirement: effective hardware prefetching. Software-managed direct memory access is more deterministic, while hardware prefetching must anticipate when data will be needed. OpenAI’s solution is to use Codex to identify kernel implementations and prefetch patterns for each shape, applying AI to the kernel-tuning process.

The Role of OpenAI’s Custom Programming Model, Gluon

The programming model called Gluon is custom-made. Built on top of OpenAI’s Triton, it follows a spatial architecture in which each physical core is programmed as a thread block. Each core has its own tensor, SIMD, and scalar engines organized around fast local memory. Gluon’s most novel abstraction is mathematical mapping between hardware resources, such as registers and warps, and tensor elements. Formalized through Linear Layouts, a type of layout algebra also invented and published by OpenAI, Gluon enables a spatial architecture in which compute is physically mapped to cores rather than logically.

Each Gluon program maps to a persistent thread, where the programmer assigns work to tiles rather than leaving that assignment entirely to the hardware scheduler. Kernels can also use hand-tuned, assembly-like code similar to NVIDIA’s PTX. Specialized collective operations handle common cross-core communication patterns, while the general network-on-chip remains available for other requirements. The kernel contains the entire inference loop, minimizing CPU involvement and orchestration steps.

The Critical Question: Will Jalapeño Replace NVIDIA?

Jalapeño is captive silicon and inference only. As such, it cannot be considered a direct competitor to NVIDIA. The software stack's maturity remains a weak point, and OpenAI is using its coding tools and AI models to iterate on kernels and port competitor models. That approach gives the company a way to improve the architecture's practical capabilities. OpenAI is expected to rely on NVIDIA solutions for training, and the two companies announced a strategic partnership that may account for up to $600 billion in compute by 2030, with NVIDIA guaranteeing OpenAI's investments.

As it stands, Jalapeño does not replace NVIDIA, yet it gives OpenAI a much-needed path to cut inference costs while providing negotiating leverage with NVIDIA (and others). OpenAI offers a portfolio of models across different price points, capabilities, latency profiles, and use cases. When an enterprise customer integrates with OpenAI's API, they typically select a specific model and build their business logic around it, and this creates a durable reality: long-lived models need long-lived infrastructure. This means OpenAI's strategy is not to replace NVIDIA with Jalapeño. It is to deploy new or high-volume models on Jalapeño where the economics clearly win, maintain existing deployments, and use the flexibility of a multi-vendor fleet to serve different performance tiers at different price points to different customers.