🖥️ Gemma 4: What Fits on Your GPU

"Frontier multimodal intelligence that actually runs on your hardware."

Four architectures, now five models covering everything from a $50 Raspberry Pi to a dual-GPU workstation.

Google DeepMind released Gemma 4 in April 2026 with an ambitious claim: these models are built from the same research as Gemini 3, their frontier closed model. The 31B Dense ranks #3 on the LMArena open model leaderboard — outperforming open models 20× its size. Since the initial release (originally four models), the family expanded to include the 12B Unified in June 2026 and MTP (Multi-Token Prediction) draft accelerators in May. For the first time, all Gemma models ship under Apache 2.0, eliminating the custom license that had frustrated the community.

But the real question isn't whether Gemma 4 is impressive on paper — it's what can you actually run, and on what hardware?

This guide maps every Gemma 4 model to real GPU configurations, shows you exactly where to download them, and tells you which distribution method is the least painful path from zero to inference.

The Gemma 4 Model Family

Gemma 4 ships in five sizes across four architectures. Understanding the architecture differences is critical — they determine what fits where.

Small Sizes (E2B, E4B): "Effective parameter" models built for edge deployment. They use Per-Layer Embeddings (PLE) to maximize parameter efficiency — each decoder layer gets its own small embedding table. The static weight footprint is larger than the effective parameter count suggests, but inference is fast because embeddings are just lookup operations. Both natively process text, image, audio, and video.

12B Unified: An encoder-free multimodal model that replaces vision and audio encoders with direct linear projections. Processes text, images, and audio natively without separate modality pipelines.

26B A4B (Mixture of Experts): 26 billion total parameters, but only 3.8 billion activate per token. All 26B must still be loaded into VRAM for fast routing — so it doesn't behave like a 4B model memory-wise. But the active compute per token is dramatically lower than a dense model.

31B Dense: The quality flagship. All 31B parameters active for every token. Maximum capability, maximum memory.

At a Glance

• Gemma 4 E2B — ~2B effective params, 128K context, text + image + audio + video
• Gemma 4 E4B — ~4B effective params, 128K context, text + image + audio + video
• Gemma 4 12B — 12B params, 256K context, text + image + video
• Gemma 4 26B A4B — 26B total / 3.8B active, 256K context, text + image + video
• Gemma 4 31B — 31B dense, 256K context, text + image + video

All models support native system prompts and function calling. MTP (Multi-Token Prediction) draft models for speculative decoding were added in May 2026 for select models.

Memory Requirements by Quantization

Based on the official HF model weights and llama.cpp quantization sizes, here are the approximate GPU memory requirements. These are base weights only — add overhead for the context window (KV cache), which scales with your prompt length.

What Fits on Your GPU

This is the practical mapping. We're looking at Q4_0 quantization (the sweet spot for most local inference — minimal quality loss, maximum compression) and 8-bit for the larger models where Q4 gets tight.

8 GB VRAM (RTX 3070, 4070, 4060 Ti)

• Gemma 4 E2B — Q4_0 at 2.9 GB. Runs comfortably with room for a substantial context window.
• Gemma 4 E4B — Q4_0 at 4.5 GB. Fits with room to spare. The most capable model that runs single-GPU on 8 GB.

The E4B at 4-bit on an 8 GB card is genuinely useful. It scores 69.4% on MMLU-Pro and handles multimodal inputs — not a toy model.

12-16 GB VRAM (RTX 4080, 4080 Super, Mac M-series)

• Gemma 4 E4B — BF16 at 17.9 GB. Tight but possible with kv-cache optimization.
• Gemma 4 12B — Q4_0 at 6.7 GB. Runs comfortably.
• Gemma 4 12B — 8-bit at 13.4 GB. Fits on 16 GB with modest context.

On a 4080 with 16 GB, you can run the 12B in 4-bit with generous context, or 8-bit for noticeably better quality with shorter prompts.

20-24 GB VRAM (RTX 3090, 4090)

This is the sweet spot for serious local inference.

• Gemma 4 12B — 8-bit at 13.4 GB. Runs with excellent quality.
• Gemma 4 26B A4B — Q4_0 at 14.4 GB. This is the killer feature. The MoE model loads entirely in 4-bit and gives you near-31B quality with only 3.8B active compute per token.
• Gemma 4 31B — Q4_0 at 17.5 GB. Fits on 24 GB. The #3 open model on LMArena, running on your desktop GPU.

32-48 GB VRAM (Dual RTX 3090, A100 40GB, Mac Studio M3 Ultra)

• Gemma 4 26B A4B — 8-bit at 28.8 GB. Runs on a single A100 or dual 3090s.
• Gemma 4 31B — 8-bit at 34.9 GB. Needs dual-GPU or a 48 GB config. Excellent quality at 8-bit.
• Gemma 4 31B — BF16 at 69.9 GB. Requires two A100s or a 48 GB rig with tensor parallelism.

On a dual-3090 rig (48 GB total), you can run the 31B in 8-bit across both GPUs with vLLM, getting near-full-precision quality with proper NVLink bandwidth.

Where to Get Them — Four Distribution Channels

Gemma 4 is available through four major distribution channels. Each has trade-offs.

Ollama

The simplest path from zero to inference. Pre-built quantizations, automatic download, built-in chat UI.

Edge models (mobile/ARM optimized):

• gemma4:e2b — Edge 2B
• gemma4:e4b — Edge 4B

Workstation models (desktop/Laptop GPUs):

• gemma4:12b — Unified 12B
• gemma4:26b — MoE 26B A4B
• gemma4:31b — Dense 31B
• gemma4:31b-cloud — 31B running on Ollama's cloud

MLX variants for Apple Silicon: gemma4:e2b-mlx, gemma4:e4b-mlx, gemma4:12b-mlx, gemma4:26b-mlx, gemma4:31b-mlx

ollama pull gemma4:31b    # downloads and stores locally
ollama run gemma4:31b     # start chatting

Ollama handles GGUF conversion, GPU detection, and layer offloading automatically. Best for getting started fast.

Hugging Face

The full ecosystem. Raw safetensors, GGUF quantizations, QAT (Quantization-Aware Training) checkpoints, compressed tensors, and community fine-tunes.

Official QAT collections:

• google/gemma-4-qat-q4-0-gguf — Drop-in GGUF for llama.cpp / LM Studio
• google/gemma-4-qat-w4a16-ct — Compressed tensors for vLLM / SGLang
• google/gemma-4-qat-q4-0-unquantized — For custom conversion or speculative decoding

QAT models are trained with quantization simulation baked in — they compensate for precision loss during training, so a 4-bit QAT model performs closer to its 16-bit baseline than a naively quantized model. Google's official recommendation for local deployment.

LM Studio

GUI-first approach. Download LM Studio, search "gemma4," pick a quantization, click "Download." It handles GGUF conversion and GPU offloading automatically.

Best for users who want a visual interface to browse quantizations, compare file sizes, and launch a chat UI without touching the command line.

Canonical Snap (Ubuntu, Linux on WSL, generic Linux)

Canonical IoT Labs published an official Gemma 4 inference snap. One command installs the E4B model with GPU acceleration:

sudo snap install gemma4
gemma4 --help

What you get: The Gemma 4 E4B multimodal model, pre-configured for NVIDIA GPU inference. Handles CUDA dependencies, model download, and runtime setup.

How it works across platforms:

• Ubuntu / Debian / Fedora / Arch (via snapd): If your distro has snapd pre-installed or easily installable, the snap runs natively.
• Windows via WSL (Windows Subsystem for Linux): Install snapd in your WSL2 Ubuntu environment, then run the same command. NVIDIA WSL drivers are supported for GPU acceleration.
• Any Linux with snapd: Snap packages are designed to be distro-agnostic — install snapd, and the snap runs identically across Ubuntu, Fedora, openSUSE, Arch (via AUR), and more.

Trade-off: Only the E4B is packaged this way — not the full family. It's the fastest path to a multimodal model with truly minimal setup friction. The snap is published by Canonical IoT Labs (verified account), last updated June 3, 2026 on the edge channel.

Sources: snapcraft.io/gemma4 · GitHub source: github.com/canonical/gemma4-snap · Bug reports: github.com/canonical/inference-snaps/issues

Which Model Should You Run?

Benchmarks That Matter

How Gemma 4 models compare at their respective sizes:

• MMLU-Pro (knowledge): E2B 60%, E4B 69.4%, 26B MoE 82.6%, 31B Dense 85.2%
• AIME 2026 (math reasoning): E2B 37.5%, E4B 42.5%, 26B MoE 88.3%, 31B Dense 89.2%
• LiveCodeBench v6 (coding): E2B 44%, E4B 52%, 26B MoE 77.1%, 31B Dense 80%
• GPQA Diamond (science): E2B 43.4%, E4B 58.6%, 26B MoE 82.3%, 31B Dense 84.3%
• MMMU Pro (multimodal): E2B 44.2%, E4B 52.6%, 26B MoE 73.8%, 31B Dense 76.9%

The gap between E4B and the MoE/31B models is significant — if your hardware can handle it, the jump to the workstation models is worth it. But the E4B is not a placeholder — it's a capable multimodal model in its own right.

The Architecture Innovations

Three architectural choices make Gemma 4 stand out:

Per-Layer Embeddings (E2B, E4B): Instead of adding more layers to increase capacity, each decoder layer gets its own embedding table. Large but cheap to load — just dictionary lookups. This is why the static memory footprint exceeds the "effective" parameter count.

Alternating Local/Global Attention (all models): Some layers use sliding-window local attention, others use full global attention. The alternation captures both fine-grained local patterns and long-range dependencies without the quadratic cost of pure global attention across 256K tokens.

Dual RoPE (all models): Two separate rotary positional embeddings — one for the backbone, one for cross-attention — enabling extended context (128K for edge models, 256K for 12B/26B/31B) without position interpolation degradation.

Fact Check Report