Runtime
llama.cpp for Gemma 4
Power-user guide to llama.cpp for Gemma 4: llama-cli and llama-server flags, Flash Attention (-fa), KV cache compression (-ctk q8_0), MoE expert offloading (-ot), and CUDA/Metal builds.
`llama.cpp` is the foundational C/C++ inference engine powering most local GGUF ecosystems—including LM Studio, Ollama, and PocketPal. Running `llama-cli` or `llama-server` directly gives you deterministic control over GPU offload, KV cache precision, batch buffers, and MoE expert routing.
For Gemma 4 specifically, staying on a current source build matters: recent upstream commits dramatically improved Gemma 4 chat template handling, multimodal projector loading, and MoE throughput.
Essential llama.cpp flags for Gemma 4 stability and speed
Combine Flash Attention with quantized KV caches and explicit batch limits when tuning tight VRAM or unified memory machines.
| Flag | Example Value | Engineering Purpose |
|---|---|---|
| -hf | ggml-org/gemma-4-E4B-it-GGUF | Pulls and caches verified official GGUF directly from Hugging Face |
| -ngl | 99 (or explicit count) | Offloads up to 99 transformer layers onto GPU VRAM / Metal |
| -fa | (flag, or auto) | Enables Flash Attention, slashing KV cache memory on long context |
| -ctk / -ctv | q8_0 | Quantizes Key and Value caches from F16 to 8-bit (~50% KV savings) |
| -c | 8192 | Caps context buffer allocation instead of defaulting to massive windows |
| -b / -ub | 512 / 256 | Controls prompt batch & micro-batch RAM spikes during prefill |
| -ot | ".ffn_.*_exps.=CPU" | Keeps shared attention in VRAM while streaming MoE experts from RAM |
Production commands: interactive server & MoE expert offload
Use `llama-server` to expose a local web UI and OpenAI-compatible API endpoint, or add `-ot` on 12–16 GB GPUs with 32–64 GB DDR5 RAM when experimenting with Gemma 4 26B-A4B MoE.
# Standard high-speed local server (24 GB GPU or 24-36 GB Mac)
llama-server -hf ggml-org/gemma-4-26B-A4B-it-GGUF:Q4_K_M \
-ngl 99 -fa on -ctk q8_0 -ctv q8_0 -c 16384 --port 8080
# Tight 12-16 GB GPU + 32 GB System RAM: keep attention & KV in VRAM, offload MoE FFN experts
llama-server -hf ggml-org/gemma-4-26B-A4B-it-GGUF:Q4_K_M \
-ngl 99 -ot ".ffn_.*_exps.=CPU" -fa on -ctk q8_0 -ctv q8_0 -c 8192Backend selection & source freshness warnings
- macOS Apple Silicon: install via `brew install llama.cpp` or build with `-DGGML_METAL=ON` so Metal unified memory activates automatically.
- NVIDIA Windows / Linux: use official prebuilt CUDA binaries or compile with `-DGGML_CUDA=ON`; avoid mismatching driver/CUDA toolkit versions (community reports have specifically flagged problematic CUDA 13.2 combinations).
- AMD Radeon / Intel Arc: Vulkan builds (`-DGGML_VULKAN=ON`) provide reliable cross-vendor GPU acceleration when ROCm or SYCL environments are unavailable.
Context shifting, prompt caching, and multimodal projectors
When running long interactive chats or agent loops via `llama-server`, prompt caching (`--cache-reuse` / slot KV reuse) avoids re-evaluating identical system prompts and repository headers across turns, cutting Time-to-First-Token from several seconds down to sub-second responses.
For multimodal Gemma 4 checkpoints (E4B, 12B, 26B-A4B), always pull both the quantized language GGUF and matching `mmproj` projector file if your repository splits them, and reserve an extra 0.5 GB to 1.2 GB of headroom for visual patch token expansion during multi-image chats.
If you serve multiple parallel editor completions or browser tabs from a single workstation GPU, tune `--parallel 2` (or `--parallel 1` on tight 12–16 GB cards) rather than leaving default multi-slot KV allocations active; each parallel slot multiplies baseline context memory overhead and can trigger unexpected CUDA or Metal allocation failures.
Launch Gemma 4 via llama-server from terminal
- Install an up-to-date `llama.cpp` release (Homebrew on macOS, winget/GitHub release on Windows, or CMake CUDA build on Linux).
- Pick a verified repository such as `ggml-org/gemma-4-E4B-it-GGUF` or `ggml-org/gemma-4-26B-A4B-it-GGUF`.
- Start `llama-server` with `-ngl 99 -fa on -ctk q8_0 -ctv q8_0 -c 8192`.
- Inspect startup stderr logs to verify `ggml_cuda` or `ggml_metal` allocated all target layers.
- Open `http://127.0.0.1:8080` in your browser or point OpenAI SDK clients to `/v1`.
Recommended models
Frequently asked questions
- Why does `--jinja` or an updated llama.cpp build matter for Gemma 4?
- Gemma 4 uses updated turn markers and reasoning/tool delimiters embedded in the GGUF Jinja template. Outdated `llama.cpp` binaries fall back to generic templates that degrade output coherence.
- Does `-ctk q8_0 -ctv q8_0` noticeably hurt model intelligence?
- In most chat and coding workloads, Q8_0 KV cache quantization matches F16 KV cache quality closely while saving up to 50% of context memory. Avoid going down to Q4_0 unless VRAM is critically tight.
- How does `-ot` help run Gemma 4 26B-A4B on a 12 GB or 16 GB GPU?
- By overriding MoE routed FFN expert tensors to CPU system RAM while pinning attention layers and KV cache in fast GPU VRAM, you avoid slow full-layer PCIe stalls and achieve usable interactive speeds.
- Should I use `llama-cli` or `llama-server` for daily usage?
- Use `llama-server` whenever you want persistent model residency in memory, a clean web chat UI, or API integration with coding tools so you do not reload multi-gigabyte weights per prompt.
- Can llama.cpp split large models across two NVIDIA GPUs?
- Yes. `llama.cpp` automatically detects multiple CUDA devices and balances layers with `--split-mode layer` and `--tensor-split` ratios.