Docs

Gemma 4 Docs

Complete technical documentation for running Gemma 4 locally: model sizing formulas, GGUF quantization tradeoffs, runtime architecture, and hardware optimization.

Running Gemma 4 locally requires matching three independent constraints: weight memory footprint, Key-Value (KV) cache growth across context length, and memory bandwidth during autoregressive token decoding.

This documentation index ties together the core engineering principles behind Gemma 4 E2B, E4B, 12B, 26B-A4B MoE, and 31B checkpoints so you can size a machine accurately before downloading multi-gigabyte model files.

Gemma 4 model family architecture

Google DeepMind shipped the Gemma 4 line under the Apache 2.0 license in two distinct architectural shapes: compact and workstation dense transformers (E2B, E4B, 12B, 31B) and a sparse Mixture-of-Experts checkpoint (26B-A4B).

Dense models activate every weight tensor on every generated token, meaning generation speed scales inversely with total quantized model size. By contrast, Gemma 4 26B-A4B holds 26B total parameters in memory (~15.8 GB at Q4_K_M) while routing only ~4B active parameters per token forward pass.

CheckpointArchitectureQ4_K_M MemoryContext WindowPrimary Target
Gemma 4 E2BDense (~5B total)~3.1 GB128KPhones, 8 GB laptops
Gemma 4 E4BDense (~8B total)~4.6 GB128KFlagship phones, 16 GB laptops
Gemma 4 12B ITUnified Multimodal Dense~7.1 GB256K12 GB GPUs, 16 GB Macs
Gemma 4 26B-A4B ITSparse MoE (4B active)~15.8 GB256K24 GB GPUs, 24–36 GB Macs
Gemma 4 31BDense (31B–33B)~19.0 GB256K24 GB+ GPUs, 32 GB+ Macs

The complete local memory equation

A frequent mistake when reading Hugging Face model cards is looking solely at the GGUF download size on disk. Actual peak RAM or VRAM required to stay responsive across multi-turn conversations follows the additive memory budget below:

Total Active Memory = Weight Footprint + KV Cache (Context x Batch x Precision) + Runtime & Shader Scratch Buffers. Leaving less than 15–20% spare headroom triggers aggressive OS swapping on macOS, silent PCIe spillover on Windows RTX cards, or instant process kills on Android and iOS.

  • Runtime & compute scratch buffers add 0.5 GB to 2.0 GB depending on batch size (-b), Metal/CUDA backend, and multimodal image patch projectors.
  • Default FP16 KV caches expand rapidly past 16K context; enabling quantized KV caches (-ctk q8_0 -ctv q8_0 or OLLAMA_KV_CACHE_TYPE=q8_0) cuts context RAM nearly in half.
  • Unified memory Macs cap safe Metal wired memory near 75% of physical RAM, whereas discrete NVIDIA GPUs trigger PCIe RAM spillover the moment VRAM exceeds 100%.

Quantization explained

Quantization compresses 16-bit floating-point (FP16 / BF16) neural weights down to 4-bit, 5-bit, 6-bit, or 8-bit integer blocks with per-block scale factors. Because local inference is overwhelmingly memory-bandwidth bound, halving weight bits doubles theoretical token throughput while halving memory pressure.

Modern K-quant formats (Q4_K_M, Q5_K_M, Q6_K) allocate extra precision bits to sensitive attention output and feed-forward tensors while compressing redundant weights more aggressively. For almost all Gemma 4 workloads, Q4_K_M is the practical baseline before trying higher bitrates.

QuantizationAvg Bits/WeightVRAM vs FP16Quality TradeoffWhen to Choose
FP16 / BF1616.0 bpw100%Zero compression lossEvaluation baselines & cloud GPUs
Q8_08.5 bpw~53%Virtually losslessAbundant VRAM / strict tool calling
Q6_K6.6 bpw~42%Near-FP16 coding & OCR24 GB+ desktops running 12B or E4B
Q4_K_M4.8 bpw~30%Best speed-to-accuracy ratioDefault starting quant on almost all hardware
Q3_K_M / IQ3_XS3.4 bpw~22%Noticeable syntax/math driftEmergency fit when Q4_K_M spills VRAM

Choosing between runtimes and backends

Pick the runtime layer based on whether you need mobile portability, a visual desktop workbench, a local API daemon, or raw llama.cpp flag tuning.

Mobile users on Android should start with AI Edge Gallery; iPhone and iPad users should use PocketPal or a Metal-compiled llama.cpp client; desktop beginners should validate hardware fit in LM Studio before wiring background services in Ollama or headless llama-server.

# Verify local hardware capacity first, then test Q4_K_M with conservative 8K context
ollama run gemma4:e4b
# Or with explicit llama.cpp Flash Attention and quantized KV cache:
llama-cli -hf ggml-org/gemma-4-E4B-it-GGUF -ngl 99 -fa -ctk q8_0 -ctv q8_0 -c 8192

Recommended local setup workflow using these docs

  1. Run the hardware compatibility checker on the homepage to inspect usable VRAM or unified memory and bandwidth.
  2. Pick the largest Gemma 4 checkpoint whose Q4_K_M footprint stays below 75% of your usable memory pool.
  3. Install the simplest runtime for your OS: LM Studio on Mac/Windows, Ollama for local APIs, or AI Edge Gallery on Android.
  4. Verify hardware acceleration is active and test a 4K–8K context conversation before increasing quantization to Q6_K or Q8_0.

Recommended models

Frequently asked questions

What is quantization in local LLMs and why does Q4_K_M matter?
Quantization packs 16-bit model weights into smaller 4-bit to 8-bit representations. Q4_K_M preserves extra precision on critical attention layers while reducing memory requirements by roughly 70%, letting 12B and 26B models run smoothly on consumer laptops and GPUs.
Why does a 15 GB GGUF file still fail to run on a 16 GB GPU or Mac?
A 15 GB file only covers static weights. You still need 2 GB to 5 GB for the KV cache, compute buffers, operating system UI, and display compositor. Staying under 75–80% memory occupancy prevents swapping and out-of-memory crashes.
Should I read the runtime docs or device guides first?
Start with your device guide (Mac, Windows, Android, or iPhone/iPad) to establish realistic model sizes for your RAM pool, then follow the matching runtime guide (LM Studio, Ollama, llama.cpp, or AI Edge Gallery) for installation flags.
Does Gemma 4 require a specific chat template in llama.cpp?
Yes. Using an outdated llama.cpp release with the wrong Gemma 4 template causes degraded reasoning, repetition, or broken tool calling. Always update llama.cpp or use GGUFs with embedded Jinja chat templates.
Where can I check non-Gemma open-weight models like Qwen 3 or DeepSeek?
Use Tool 2 on the homepage or the Compare page to test 130+ open-weight LLMs including Qwen 3, Llama 3.3, DeepSeek-R1, and Mistral against the same hardware sizing rules.