OpenAI’s Jalapeño Chip Delivers Faster, Cheaper AI Inference

OpenAI on Aug. 25 released the first hard numbers for Jalapeño, its custom-designed AI inference chip, showing it substantially outperforms leading Nvidia rack systems on the two metrics that matter most at scale: how much useful AI work the hardware can complete per watt of power, and how quickly it delivers responses to users.

The results, presented at the Hot Chips conference, were measured using InferenceX, a public benchmark from research firm SemiAnalysis that simulates the full lifecycle of serving an AI request. Tested across three publicly available models — GPT-OSS 120B, DeepSeek R1 670B and Kimi K2.5 1T — Jalapeño delivered between 1.5x and 1.9x more AI work per watt at peak throughput, and between 1.7x and 3.6x lower end-to-end latency, compared to Nvidia’s GB200 and GB300 systems. For agentic workloads, where many inference steps chain together in sequence, performance jumped even higher: 2.1x to 4.1x better than the comparison hardware.

OpenAI’s core architectural claim is that Jalapeño sidesteps a tradeoff that has plagued inference hardware design. Most systems are tuned either for high throughput — handling many requests at once — or for low latency — returning any single response quickly. Jalapeño, the company says, achieves both simultaneously by designing the chip, memory, networking and software stack as a single integrated system rather than assembling off-the-shelf components.

“Jalapeño delivers both higher throughput and lower latency with one architecture, where existing hardware systems often have to make a tradeoff between the two,” wrote OpenAI.

The chip is rated at 700 watts but sustained measured power during testing stayed at or below 550 watts — a gap that matters when efficiency figures are normalized to power ratings. On the largest model tested, Kimi K2.5 1T, Jalapeño delivered approximately 1.5x higher peak performance per watt and 3.4x lower end-to-end latency than the comparison system. OpenAI also says its advantage grew further on internal frontier models, suggesting the architecture scales well as model size increases.

How Jalapeño Was Built — and How AI Helped Build It

Jalapeño went from initial design to tapeout in nine months, a pace OpenAI credits partly to using its own AI models throughout the development process. Earlier generations of OpenAI models helped the engineering team explore chip designs and shorten verification cycles; more recent models are now accelerating how the chip is optimized and programmed for new workloads.

Using Codex alongside a model called GPT-Astra, the team brought three open-weight models — models that were not part of Jalapeño’s original production plan — to high performance in about two months. For selected attention and mixture-of-experts computational blocks, AI-generated code ran between 1.5x and 1.8x faster than implementations written by human experts. Those figures apply to specific blocks, not full model runs, but they point toward a meaningful shift in how chip software gets written.

Jalapeño was built in partnership with Broadcom, which handled silicon implementation, networking and connectivity. Celestica contributed board, rack and systems integration expertise. OpenAI head of hardware Richard Ho has said initial deployment will happen in late 2026 in small volumes, with more significant rollout in 2027. Generations 2 and 3 of the platform are already in development.

A Crowded Market — With an Important Caveat

Jalapeño enters a competitive field. Google’s TPU v7 Ironwood, Microsoft’s Maia 200, Amazon’s Trainium 3 and Meta’s MTIA all target the inference workload that now represents roughly two-thirds of total AI compute, up from one-third in 2023, according to Deloitte projections. But there is a key distinction worth understanding: all of these chips, Jalapeño included, are captive infrastructure. They will serve their respective company’s own products and APIs. None of them will be available for external developers to rent or buy.

For students and developers building on GPU cloud, the accessible chip landscape remains dominated by Nvidia, with AMD as a distant alternative. Nvidia’s next-generation Rubin GPUs and AMD’s MI455X are both expected to ramp production in early 2027. Unlike Jalapeño, those chips are optimized for a mix of training and inference workloads, while Jalapeño focuses exclusively on inference — a specialization that yields efficiency gains but also introduces architectural risk if OpenAI’s model designs change significantly in future generations.

What It Means for Students and Developers

Jalapeño is not something you can access directly. But it will power the infrastructure behind ChatGPT, the OpenAI API and tools like Codex — which means its performance improvements can show up in tangible ways for people building on those platforms.

Lower inference costs historically translate, at least partially, into lower API pricing for developers. For students using the OpenAI API for class projects, portfolio apps or early-stage products, that can mean more affordable per-token costs over time and more reliable access during high-demand periods. Faster hardware also enables faster, more capable agentic workflows — the kind that can complete multi-step tasks without accumulating delays at every inference call.

“For customers, that can mean faster responses, more responsive agents, and more reliable access as demand grows,” wrote OpenAI.

For students interested in AI hardware careers, the development story behind Jalapeño signals an emerging discipline. A nine-month chip design cycle, accelerated by AI-generated code across the compiler and kernel layers, blends skills from ML engineering, chip architecture and systems programming — a combination that is increasingly in demand across labs and semiconductor companies.

OpenAI plans to begin deploying Jalapeño within its compute infrastructure before the end of 2026, with broader rollout through 2027 and beyond. The company describes it as the first step in a multi-generational silicon platform — meaning the efficiency and latency gains announced this week are intended as a floor, not a ceiling.

Source: OpenAI