LLamjinLab
AI NewsWords 1522Read time≈ 4 min

OpenAI’s First Jalapeño Benchmarks Claim Lower Latency and Higher Efficiency Than Nvidia

OpenAI says its Jalapeño inference chip beats Nvidia GB200 and GB300 systems on latency and performance per watt in early InferenceX tests.

AILLMOpenAIAgent

OpenAI published the first measured performance results for Jalapeño, its custom AI inference chip, on August 25, 2026. Across three public language models, the company reported 1.5 to 1.9 times more peak throughput per watt and 1.7 to 3.6 times lower end-to-end latency than the Nvidia systems used for comparison.

The results move Jalapeño beyond the projections OpenAI and Broadcom presented when they unveiled the processor on June 24. At that point, engineering samples were operating at their target frequency and power, but OpenAI had not released detailed performance figures. The new tests cover GPT‑OSS 120B, DeepSeek R1 670B and Kimi K2.5 1T.

Jalapeño remains pre-production hardware rather than a product available to customers. OpenAI plans to begin deploying a limited number of systems inside its own infrastructure by the end of 2026 while it completes production qualification, improves the software stack and validates more workloads.

1. What the first Jalapeño benchmarks show

OpenAI tested Jalapeño using InferenceX, SemiAnalysis’ public suite for measuring language-model serving systems. Unlike a single theoretical compute figure, these tests examine the relationship between throughput and the latency experienced by each user.

The published comparisons use a nominal workload of 8,000 input tokens and 1,000 output tokens with single-token prediction. OpenAI normalized performance using each accelerator’s published package power rating: 700 watts for Jalapeño, 1,200 watts for the Nvidia GB200 and 1,400 watts for the GB300. OpenAI said Jalapeño’s measured sustained power stayed at or below 550 watts during the tested workloads.

On GPT‑OSS 120B, Jalapeño was compared with GB200. OpenAI reported approximately 85,448 mixed tokens per second per kilowatt at peak throughput, against 44,960 for the Nvidia system—a 1.9-fold advantage. End-to-end latency was 1.03 seconds rather than 1.80 seconds, while minimum time between generated tokens was 0.69 milliseconds versus 1.87 milliseconds.

The DeepSeek R1 670B comparison used GB300. Jalapeño produced 19,641 mixed tokens per second per kilowatt at peak throughput, compared with 11,781, or about 1.7 times as much. Its reported end-to-end latency was 1.65 seconds, versus 5.99 seconds, and its minimum time between tokens was 1.43 milliseconds rather than 5.90 milliseconds.

For Kimi K2.5 1T, also tested against GB300, Jalapeño reached 18,195 mixed tokens per second per kilowatt, compared with 11,862. End-to-end latency was 1.56 seconds rather than 5.31 seconds, while minimum time between tokens fell from 5.48 milliseconds to 1.44 milliseconds.

OpenAI also presented large throughput gains when each system was constrained to the previous competitor’s best time-between-tokens point. Those figures reached 53.7 times on GPT‑OSS, 104.3 times on DeepSeek R1 and 56.1 times on Kimi K2.5. These are matched-interactivity measurements at particular operating points, not claims that every Jalapeño workload is tens of times faster.

Across the broader test range, OpenAI summarized Jalapeño’s advantage on highly interactive workloads as 2.1 to 4.1 times. The chip placed on the throughput-per-kilowatt Pareto frontier for all three models, meaning none of the compared configurations simultaneously offered greater throughput and lower latency.

2. The results are promising but not full production validation

SemiAnalysis developed InferenceX and observed the Jalapeño runs with OpenAI engineers in the company’s laboratory. It confirmed that the benchmarked models produced GSM8K evaluation results comparable to those obtained on Nvidia hardware, reducing the risk that speed had been achieved by silently compromising model output.

However, SemiAnalysis stressed that “all numbers are provided to us by OpenAI.” Its analysts witnessed the runs but did not execute the entire InferenceX suite independently. They also have not seen Jalapeño results on AgentX, a newer benchmark designed around long-context, multi-turn agent workloads.

That distinction matters because the published 8K-input, 1K-output tests do not fully exercise production components such as request routers, prefix caches, cache-management systems and offload infrastructure. Performance on a carefully optimized single-turn workload may not translate directly to a live service handling variable prompts, sustained traffic and multi-step agents.

The comparison generation also requires context. GPT‑OSS was measured against GB200, while the larger DeepSeek and Kimi models were compared with GB300. SemiAnalysis argues that Nvidia’s newer Vera Rubin platform is the more relevant competitive reference because both Rubin and Jalapeño use HBM4-class memory technology and are at relatively early stages of software development.

The tests therefore establish that working Jalapeño engineering samples can run several large, non-proprietary model families efficiently. They do not yet establish fleet-level reliability, total cost of ownership or performance under ChatGPT and API production traffic.

3. How Jalapeño targets both latency and throughput

Jalapeño is an inference-only accelerator rather than a chip for training new foundation models. OpenAI designed it with Broadcom around the distinct phases of serving a language model.

During prefill, the system processes the user’s prompt, making compute capacity a central constraint. During decode, it generates the answer token by token, placing more pressure on memory bandwidth. Moving model state between processors can add another bottleneck, particularly when a request spans multiple chips.

OpenAI says Jalapeño reduces this movement by keeping model state—including the key-value cache used during generation—local when possible. Compute, memory and networking are coordinated within a large connected domain, allowing the same accelerator pool to handle both prefill and decode rather than assigning those phases permanently to separate hardware.

That design is intended to preserve efficiency as workload ratios change. A batch-oriented service may prioritize total throughput, while an interactive agent may need rapid token generation because delays accumulate across sequential tool calls and reasoning steps.

Broadcom supplied silicon implementation, connectivity and networking expertise, including its Tomahawk technology. Celestica is contributing board, rack and system integration. OpenAI remains responsible for the architecture and the software-hardware decisions informed by its ChatGPT, Codex and API workloads.

The company has not announced plans to sell Jalapeño chips or systems. OpenAI hardware chief Richard Ho told Axios that internal demand is already too large for the company to envision supplying outside buyers. Developers would therefore encounter Jalapeño indirectly through OpenAI services rather than as a purchasable accelerator.

4. AI contributed to both chip design and software bring-up

OpenAI says AI tools participated directly in Jalapeño’s development. From initial design to manufacturing tape-out, the company completed the chip-design phase in nine months. SemiAnalysis places the wider timeline—from initial team hiring to tape-out—at approximately 16 months, so the two figures describe different starting points rather than necessarily conflicting schedules.

Models were used to explore implementations, shorten design and verification loops, and optimize arithmetic circuits. OpenAI also structured the processor as a predictable programming target built around local tensors, explicit communication and synchronization, allowing AI systems to help place and schedule work across the hardware.

Using Codex with GPT‑Astra, the engineering team brought GPT‑OSS, DeepSeek R1 and Kimi K2.5—none of which had been in Jalapeño’s original production plan—to high performance within two months. For selected GPT‑OSS attention and mixture-of-experts blocks, AI-generated implementations ran 1.5 to 1.8 times faster than existing code written by human experts. OpenAI explicitly limits that result to those blocks; it is not a full-model speedup.

OpenAI expects limited Jalapeño deployments by the end of 2026 and broader capacity during 2027. A second-generation design is already in advanced development, while work on a third generation has begun.

The company is not replacing its external suppliers. It says Nvidia and other partners will continue providing accelerators for training and inference because Jalapeño alone cannot satisfy its compute requirements. The immediate operational test is whether OpenAI can reproduce the laboratory efficiency gains after production qualification and deployment across full racks under mixed customer traffic.

Frequently Asked Questions

What is OpenAI Jalapeño?

Jalapeño is OpenAI’s first custom AI accelerator. It was co-developed with Broadcom specifically for language-model inference rather than model training.

Is Jalapeño faster than Nvidia’s chips?

In OpenAI’s published InferenceX tests, Jalapeño delivered higher peak throughput per kilowatt and lower latency than the GB200 and GB300 configurations compared. The results come from engineering samples and are not yet full production benchmarks.

Which models were tested?

OpenAI tested GPT‑OSS 120B, DeepSeek R1 670B and Kimi K2.5 1T using nominal 8,000-input-token and 1,000-output-token workloads.

Can developers buy or rent Jalapeño hardware?

No. OpenAI says it has no plans to sell the chip because it needs the available capacity for its own infrastructure. Developers may eventually benefit through OpenAI-hosted products and APIs.

When will Jalapeño enter service?

OpenAI plans to begin limited internal deployment by the end of 2026. Production qualification, software development and testing across additional models are still underway.

Sources

Share

Share this article