Google Brings PyTorch to TPUs With New vLLM and SGLang Engines

Google on Oct. 20, 2026 announced native PyTorch support for two of the most widely used open-source LLM inference frameworks — vLLM and SGLang — allowing them to run directly on Google’s Tensor Processing Units without requiring JAX, the framework developers previously had to work through. The launch, part of a broader set of PyTorch-related announcements, centers on a new runtime layer called TorchTPU that treats TPUs as standard PyTorch devices, meaning tensors, compilation via torch.compile, and collective operations all behave the way PyTorch developers already expect.

The two resulting engines — vLLM-TorchTPU, built in partnership with Inferact, and SGLang-TorchTPU, built with RadixArk — were co-developed with the companies founded by the original creators of each framework. Both are already running in production with preview customers. SGLang-TorchTPU is live at Meta at scale. Performance numbers are being published publicly through the SemiAnalysis InferenceX and AgentX leaderboards, with Google reporting a cost of 2.01 cents per million tokens on its Ironwood TPU (the seventh-generation TPU, also called TPU v7) — a figure the company is calling industry-leading.

What’s Actually New Here

Before this release, vLLM and SGLang could already run on TPUs, but only through JAX-based backends that required developers to step outside the PyTorch mental model they use everywhere else. The complaint Google kept hearing was that engineers wanted to customize and debug their serving stack in PyTorch — not learn a second framework just to access TPU hardware. TorchTPU addresses that directly: tensors are managed as standard PyTorch tensors on a tpu device, code runs eagerly or compiles through XLA, and there are no wrapper types to navigate.

A notable technical detail: a custom Pallas megakernel for the 2.8-trillion-parameter Kimi K3 model runs the entire decode step as a single kernel, delivering more than 700 tokens per second to a single user on Ironwood TPU. KV cache transfers between hosts are handled by an open-source C++ library called TPU Sync, which reads device buffers directly to avoid staging copies and runs transfers on separate threads so they don’t block inference computation.

“Our goal is for the PyTorch community to bring models to TPU and make them fast using the tools developers are already familiar with,” Google wrote in the announcement.

Supported model families at launch include Qwen, DeepSeek, GLM, Kimi, GPT-OSS and Gemma4, with day-zero support for new open-weight models planned. The existing JAX-based backends, vLLM-JAX and SGLang-JAX, will continue to be maintained alongside the new PyTorch-native ones.

Where This Fits in a Competitive Market

The LLM inference engine space is effectively a three-way race between vLLM, SGLang and Nvidia’s TensorRT-LLM. On an H100 with Llama 3.3 70B at 50 concurrent requests, throughput benchmarks show TensorRT-LLM slightly ahead of SGLang, which edges out vLLM. But in prefix-heavy workloads — shared system prompts, retrieval-augmented generation, multi-turn agent conversations — SGLang has shown meaningfully lower time-to-first-token than vLLM, which matters for the agentic use cases Google is explicitly targeting with this launch.

The bigger competitive gap this announcement closes is on hardware reach. vLLM already supports AMD ROCm, Intel Gaudi and AWS Neuron in addition to Nvidia GPUs. SGLang, until now, has been largely limited to Nvidia and AMD hardware. This release gives SGLang its first production-grade TPU path — a significant expansion in the places developers can deploy it.

The Student Angle: Free TPU Compute, Tied to Actual Contributions

For students and early-career engineers looking to break into ML infrastructure — one of the highest-demand specializations in AI engineering — the most practically significant piece of this announcement may be a separate but simultaneous launch: a community contribution TPU capacity pool that grants free TPU compute to developers who actively contribute code to either engine.

“vLLM is also launching a first-of-its-kind community contribution TPU capacity pool to provide free TPU compute to developers who actively contribute to both engines,” the announcement states.

TPU experience has historically been nearly impossible to acquire outside of Google itself or large enterprise cloud customers. That has made it a genuine resume differentiator — and a genuine barrier. The contributor pool changes the calculus by tying access to shipping code rather than submitting a project proposal, which is how competing programs like AWS’s researcher credit program or Nvidia’s Academic Grant Program typically work. There is no publicly stated hours ceiling on the Google pool, and Inferact’s maintainers will help with design and pull-request reviews for contributors who receive access, adding a mentorship layer that pure credit programs don’t offer.

Crucially, the barrier to entry for contributions is lower than it might sound. Both engines preserve the standard vLLM and SGLang developer experience — same CLI, same OpenAI-compatible API server, same model definitions — with TPU optimizations added underneath. A student already familiar with either framework from coursework or side projects can contribute without learning an entirely new stack. An approved pull request to vllm-torchtpu or sglang-torchtpu is a public, measurable contribution to some of the most-watched open-source AI repositories in existence, with a CI pipeline that benchmarks and publishes support for new open-weight models within 24 hours of release.

For anyone building toward a career in AI infrastructure, that combination — public GitHub contributions, TPU hardware access and direct feedback from the engineers who built the frameworks — is worth paying attention to. The contribution pool request form is linked from the official announcement.

Source: Google AI