Google Runs Gemma AI Offline on Raspberry Pi 5 With LiteRT

Google AI Edge published a developer guide this week demonstrating something that would have seemed impractical just a year ago: running a capable large language model fully offline on a Raspberry Pi 5 — the $80 single-board computer that students and hobbyists have used for robotics and tinkering projects for over a decade. The toolchain involves two Google-built components, LiteRT and Gemma 4, and the bar to get started is now a single pip install.

What Google Actually Announced

The announcement centers on LiteRT, Google’s on-device inference runtime (evolved from TensorFlow Lite), and a specialized orchestration layer on top of it called LiteRT-LM, which handles LLM deployment specifically. Together, they let developers run Gemma 4 E2B — one of the smallest models in Google’s open Gemma family — directly on a Pi 5’s quad-core ARM Cortex-A76 processor, no internet connection required.

The performance numbers are more impressive than the hardware might suggest. On a Raspberry Pi 5, LiteRT-LM achieves 99 tokens per second for prefill and 9 tokens per second for decode while running Gemma 4 E2B, with a peak memory footprint of just 1,432 MB. Because Gemma 4 E2B’s tokenizer packs roughly 4.2 characters into each token, that decode rate translates to an end-to-end generation speed of approximately 27.3 characters per second — about 300 words per minute, or twice the pace of normal human speech.

To show what that means in practice, Google demoed a Reachy Mini robot powered entirely by a Raspberry Pi 5. The robot can see, hear and respond to its environment in real time using a split processing architecture: LLM inference runs on the CPU, while vision and audio models are offloaded to the Pi’s Broadcom VideoCore VII GPU via LiteRT’s WebGPU (Vulkan) backend through a component called ML Drift. The CPU delivers roughly 153.6 GFLOPS at FP32 precision; the GPU offers about 76.8 GFLOPS at the same precision but adds parallel execution that keeps the CPU free for higher-priority tasks. Supported GPU models include MediaPipe pipelines, Ultralytics YOLO object detection and Moonshine audio transcription.

Google also released a new LiteRT CLI tool that consolidates model conversion, quantization, benchmarking and inference into a single command set. The full workflow — from install to running Gemma 4 E2B — is designed to take just a few terminal commands. Separately, Google teased that LiteRT and Gemma support is coming soon to Hailo AI accelerators, including the Raspberry Pi AI HAT+ 2, an add-on board that would substantially accelerate inference beyond what the Pi’s built-in silicon can handle.

How This Fits Into a Crowded Field

Edge AI runtimes are not a new category. Ollama has made it easy to run quantized models on consumer hardware, but it was designed primarily for desktop and server machines and runs llama.cpp under the hood — not purpose-built for the tight memory and thermal budgets of an IoT device. ONNX Runtime is generally faster than LiteRT’s predecessor (TensorFlow Lite) on the same Cortex-A76 silicon and is often the preferred choice for YOLO and PyTorch-origin models. Meta’s ExecuTorch and llama.cpp itself round out a runtime landscape that is genuinely competitive.

Where LiteRT and Gemma 4 make a distinct argument is in native multimodal capability on constrained hardware. Gemma 4’s edge models handle audio and vision out of the box, without requiring separate pipeline components. Competing models in the same parameter range — Microsoft’s Phi-3 or Alibaba’s Qwen2.5, for instance — often outperform Gemma on pure text reasoning benchmarks but need additional models bolted on for audio and vision tasks. For developers building robotics or voice-first applications on a Pi, that integration simplifies the architecture considerably.

Why Students Should Pay Attention

The practical implication for students and recent grads is straightforward: you can now build a portfolio-worthy AI project with hardware that costs less than a textbook. An offline voice assistant, a real-time object-detection robot, a local translation tool — any of these can run on a Raspberry Pi 5 with 8 GB of RAM, with no API fees and no data leaving the device. The open-source Gemma Translator repository and the Reachy Mini demo code, both linked from the Google announcement, provide concrete starting points rather than blank-page anxiety.

The LiteRT CLI also removes one of the most common stumbling blocks for beginners: environment setup. Consolidating conversion, quantization and inference into one tool means less time debugging dependency conflicts and more time actually building.

More broadly, the skills on display here — deploying AI at the edge, designing for memory and latency constraints, splitting workloads across CPU and GPU — are increasingly valued in embedded systems, robotics and IoT job roles. Building something that works offline, protects user privacy and operates under tight hardware limits is a different design challenge than calling an API, and employers working on devices that cannot rely on a cloud connection know the difference.

The upcoming Hailo HAT+ 2 integration is worth watching as well. If it delivers on Google’s claim of “massive hardware acceleration” through the same familiar LiteRT workflows, the performance ceiling for student projects on Pi hardware will rise significantly without requiring new code.

Source: Google