Start

Getting Started with Gemma 4

Step-by-step onboarding from hardware check to your first verified local Gemma 4 chat: choosing a checkpoint, confirming GPU offload, and scaling context safely.

Most failed first-time local AI setups share the same root cause: starting from a leaderboard chart, downloading the largest checkpoint available, and blaming the runtime when a 16 GB machine stalls or crashes.

Following a structured verification sequence gets you to a fast, reliable local setup in under 15 minutes—and ensures you immediately catch misconfigured GPU drivers or memory spillover before debugging prompts.

Phase 1: Size your hardware honestly before downloading

Every device falls into a predictable local AI capacity band determined by dedicated GPU VRAM (on Windows/Linux) or unified/shared RAM (on Apple Silicon and phones). Never size against total disk space or marketing "shared graphics memory".

Device ProfileFirst Checkpoint to Prove PipelineDaily Upgrade TargetFirst Runtime to Try
Android Phone (8–12 GB)Gemma 4 E2BGemma 4 E4B (12–16 GB RAM)AI Edge Gallery
iPhone / iPad (8 GB)Gemma 4 E2BGemma 4 E4B (16 GB iPad)PocketPal
Windows Laptop (8 GB VRAM)Gemma 4 E4BGemma 4 12B (short context)LM Studio
Apple Silicon Mac (16 GB)Gemma 4 E4BGemma 4 12B / 26B-A4BLM Studio or Ollama
RTX Desktop (24 GB VRAM)Gemma 4 12BGemma 4 26B-A4B / 31BLM Studio or llama.cpp

Phase 2: Run a minimal 4K smoke test and verify acceleration

After installing your chosen runtime and pulling Q4_K_M weights, test a short single-turn text prompt at 4,096 context tokens. Do not start by pasting a 30,000-token codebase or enabling experimental tool-calling plugins.

Immediately verify that hardware acceleration is actually executing the forward pass. Look for 100% GPU residency in `ollama ps`, Dedicated VRAM climbing cleanly in Windows Task Manager, or GPU History spikes in macOS Activity Monitor.

  • If your first prompt outputs under 3 tokens per second on a desktop GPU, stop and fix PCIe offloading or missing CUDA/Vulkan backends before testing anything larger.
  • If load hangs or exits immediately on mobile, close background apps and step down from E4B to E2B.
  • Once short text replies stream smoothly, enable Flash Attention and Q8_0 KV cache quantization before pushing context to 16K–32K.

Phase 3: Unlock multimodal vision, coding tools, and APIs

Gemma 4 checkpoints (especially E4B, 12B IT, and 26B-A4B IT) natively support image inputs, structured reasoning, and agent tool calls—provided your runtime supplies the current Gemma 4 Jinja chat template.

Once your baseline chat feels effortless, you can wire Ollama (`http://localhost:11434/v1`) or LM Studio’s local server (`http://localhost:1234/v1`) into coding assistants such as Continue, Roo Code, Open WebUI, or local Python scripts.

Treat tool calling and strict JSON output as your final graduation test: if a Q4_K_M checkpoint occasionally drops a JSON bracket on long schema prompts, keep the same model size and step quantization up to Q6_K or Q8_0 before jumping to a heavier parameter count.

# 1. Start with a fast baseline pull in Ollama
ollama run gemma4:e4b

# 2. Confirm 100% GPU offload in a second shell
ollama ps

# 3. Call the local OpenAI-compatible chat endpoint
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma4:e4b","messages":[{"role":"user","content":"Summarize local KV cache tradeoffs in 3 bullets."}]}'

Go from zero to working local Gemma 4 in 5 steps

  1. Run the hardware compatibility checker on the homepage to identify your device memory pool.
  2. Pick the smallest recommended Gemma 4 checkpoint for your tier (E2B on phones, E4B or 12B on 16 GB desktops).
  3. Install LM Studio (GUI) or Ollama (CLI/API) on desktop, or AI Edge Gallery on Android.
  4. Download Q4_K_M weights and run one short prompt at 4K–8K context.
  5. Verify 100% GPU or Metal acceleration before expanding context length or enabling tool calling.

Recommended models

Frequently asked questions

Which Gemma 4 file format should beginners download first?
Start with a GGUF build at Q4_K_M quantization (or native LiteRT inside AI Edge Gallery on Android). Q4_K_M balances strong answer quality with ~70% smaller memory needs than FP16.
Should I use Base or IT (Instruction-Tuned) Gemma 4 checkpoints?
Always pick IT (instruction-tuned) checkpoints such as gemma4-e4b-it or gemma4-26b-a4b-it for chat, coding, vision, and assistant workflows. Base checkpoints are intended for custom fine-tuning and raw text completion.
How can I verify my machine is actually offline during inference?
Once model weights finish downloading in LM Studio, Ollama, or PocketPal, disconnect Wi-Fi or block outbound access—every prompt and generation runs 100% locally on your CPU/GPU.
What if Gemma 4 feels slower than an online browser chat?
Browser playgrounds run on datacenter GPUs. If your local speed sits below 10 tokens/sec, check whether layers spilled out of VRAM or step down one model size (for example 26B to 12B or E4B).
Do I need Python or coding knowledge to run Gemma 4 locally?
No. GUI apps like LM Studio on Mac/Windows and AI Edge Gallery or PocketPal on mobile let you search, download, and chat with zero command-line setup.