Docs
Gemma 4 Docs
Complete technical documentation for running Gemma 4 locally: model sizing formulas, GGUF quantization tradeoffs, runtime architecture, and hardware optimization.
Running Gemma 4 locally requires matching three independent constraints: weight memory footprint, Key-Value (KV) cache growth across context length, and memory bandwidth during autoregressive token decoding.
This documentation index ties together the core engineering principles behind Gemma 4 E2B, E4B, 12B, 26B-A4B MoE, and 31B checkpoints so you can size a machine accurately before downloading multi-gigabyte model files.
Gemma 4 model family architecture
Google DeepMind shipped the Gemma 4 line under the Apache 2.0 license in two distinct architectural shapes: compact and workstation dense transformers (E2B, E4B, 12B, 31B) and a sparse Mixture-of-Experts checkpoint (26B-A4B).
Dense models activate every weight tensor on every generated token, meaning generation speed scales inversely with total quantized model size. By contrast, Gemma 4 26B-A4B holds 26B total parameters in memory (~15.8 GB at Q4_K_M) while routing only ~4B active parameters per token forward pass.
| Checkpoint | Architecture | Q4_K_M Memory | Context Window | Primary Target |
|---|---|---|---|---|
| Gemma 4 E2B | Dense (~5B total) | ~3.1 GB | 128K | Phones, 8 GB laptops |
| Gemma 4 E4B | Dense (~8B total) | ~4.6 GB | 128K | Flagship phones, 16 GB laptops |
| Gemma 4 12B IT | Unified Multimodal Dense | ~7.1 GB | 256K | 12 GB GPUs, 16 GB Macs |
| Gemma 4 26B-A4B IT | Sparse MoE (4B active) | ~15.8 GB | 256K | 24 GB GPUs, 24–36 GB Macs |
| Gemma 4 31B | Dense (31B–33B) | ~19.0 GB | 256K | 24 GB+ GPUs, 32 GB+ Macs |
The complete local memory equation
A frequent mistake when reading Hugging Face model cards is looking solely at the GGUF download size on disk. Actual peak RAM or VRAM required to stay responsive across multi-turn conversations follows the additive memory budget below:
Total Active Memory = Weight Footprint + KV Cache (Context x Batch x Precision) + Runtime & Shader Scratch Buffers. Leaving less than 15–20% spare headroom triggers aggressive OS swapping on macOS, silent PCIe spillover on Windows RTX cards, or instant process kills on Android and iOS.
- Runtime & compute scratch buffers add 0.5 GB to 2.0 GB depending on batch size (-b), Metal/CUDA backend, and multimodal image patch projectors.
- Default FP16 KV caches expand rapidly past 16K context; enabling quantized KV caches (-ctk q8_0 -ctv q8_0 or OLLAMA_KV_CACHE_TYPE=q8_0) cuts context RAM nearly in half.
- Unified memory Macs cap safe Metal wired memory near 75% of physical RAM, whereas discrete NVIDIA GPUs trigger PCIe RAM spillover the moment VRAM exceeds 100%.
Quantization explained
Quantization compresses 16-bit floating-point (FP16 / BF16) neural weights down to 4-bit, 5-bit, 6-bit, or 8-bit integer blocks with per-block scale factors. Because local inference is overwhelmingly memory-bandwidth bound, halving weight bits doubles theoretical token throughput while halving memory pressure.
Modern K-quant formats (Q4_K_M, Q5_K_M, Q6_K) allocate extra precision bits to sensitive attention output and feed-forward tensors while compressing redundant weights more aggressively. For almost all Gemma 4 workloads, Q4_K_M is the practical baseline before trying higher bitrates.
| Quantization | Avg Bits/Weight | VRAM vs FP16 | Quality Tradeoff | When to Choose |
|---|---|---|---|---|
| FP16 / BF16 | 16.0 bpw | 100% | Zero compression loss | Evaluation baselines & cloud GPUs |
| Q8_0 | 8.5 bpw | ~53% | Virtually lossless | Abundant VRAM / strict tool calling |
| Q6_K | 6.6 bpw | ~42% | Near-FP16 coding & OCR | 24 GB+ desktops running 12B or E4B |
| Q4_K_M | 4.8 bpw | ~30% | Best speed-to-accuracy ratio | Default starting quant on almost all hardware |
| Q3_K_M / IQ3_XS | 3.4 bpw | ~22% | Noticeable syntax/math drift | Emergency fit when Q4_K_M spills VRAM |
Choosing between runtimes and backends
Pick the runtime layer based on whether you need mobile portability, a visual desktop workbench, a local API daemon, or raw llama.cpp flag tuning.
Mobile users on Android should start with AI Edge Gallery; iPhone and iPad users should use PocketPal or a Metal-compiled llama.cpp client; desktop beginners should validate hardware fit in LM Studio before wiring background services in Ollama or headless llama-server.
# Verify local hardware capacity first, then test Q4_K_M with conservative 8K context
ollama run gemma4:e4b
# Or with explicit llama.cpp Flash Attention and quantized KV cache:
llama-cli -hf ggml-org/gemma-4-E4B-it-GGUF -ngl 99 -fa -ctk q8_0 -ctv q8_0 -c 8192Recommended local setup workflow using these docs
- Run the hardware compatibility checker on the homepage to inspect usable VRAM or unified memory and bandwidth.
- Pick the largest Gemma 4 checkpoint whose Q4_K_M footprint stays below 75% of your usable memory pool.
- Install the simplest runtime for your OS: LM Studio on Mac/Windows, Ollama for local APIs, or AI Edge Gallery on Android.
- Verify hardware acceleration is active and test a 4K–8K context conversation before increasing quantization to Q6_K or Q8_0.
Recommended models
Frequently asked questions
- What is quantization in local LLMs and why does Q4_K_M matter?
- Quantization packs 16-bit model weights into smaller 4-bit to 8-bit representations. Q4_K_M preserves extra precision on critical attention layers while reducing memory requirements by roughly 70%, letting 12B and 26B models run smoothly on consumer laptops and GPUs.
- Why does a 15 GB GGUF file still fail to run on a 16 GB GPU or Mac?
- A 15 GB file only covers static weights. You still need 2 GB to 5 GB for the KV cache, compute buffers, operating system UI, and display compositor. Staying under 75–80% memory occupancy prevents swapping and out-of-memory crashes.
- Should I read the runtime docs or device guides first?
- Start with your device guide (Mac, Windows, Android, or iPhone/iPad) to establish realistic model sizes for your RAM pool, then follow the matching runtime guide (LM Studio, Ollama, llama.cpp, or AI Edge Gallery) for installation flags.
- Does Gemma 4 require a specific chat template in llama.cpp?
- Yes. Using an outdated llama.cpp release with the wrong Gemma 4 template causes degraded reasoning, repetition, or broken tool calling. Always update llama.cpp or use GGUFs with embedded Jinja chat templates.
- Where can I check non-Gemma open-weight models like Qwen 3 or DeepSeek?
- Use Tool 2 on the homepage or the Compare page to test 130+ open-weight LLMs including Qwen 3, Llama 3.3, DeepSeek-R1, and Mistral against the same hardware sizing rules.