Start
Getting Started with Gemma 4
Step-by-step onboarding from hardware check to your first verified local Gemma 4 chat: choosing a checkpoint, confirming GPU offload, and scaling context safely.
Most failed first-time local AI setups share the same root cause: starting from a leaderboard chart, downloading the largest checkpoint available, and blaming the runtime when a 16 GB machine stalls or crashes.
Following a structured verification sequence gets you to a fast, reliable local setup in under 15 minutes—and ensures you immediately catch misconfigured GPU drivers or memory spillover before debugging prompts.
Phase 1: Size your hardware honestly before downloading
Every device falls into a predictable local AI capacity band determined by dedicated GPU VRAM (on Windows/Linux) or unified/shared RAM (on Apple Silicon and phones). Never size against total disk space or marketing "shared graphics memory".
| Device Profile | First Checkpoint to Prove Pipeline | Daily Upgrade Target | First Runtime to Try |
|---|---|---|---|
| Android Phone (8–12 GB) | Gemma 4 E2B | Gemma 4 E4B (12–16 GB RAM) | AI Edge Gallery |
| iPhone / iPad (8 GB) | Gemma 4 E2B | Gemma 4 E4B (16 GB iPad) | PocketPal |
| Windows Laptop (8 GB VRAM) | Gemma 4 E4B | Gemma 4 12B (short context) | LM Studio |
| Apple Silicon Mac (16 GB) | Gemma 4 E4B | Gemma 4 12B / 26B-A4B | LM Studio or Ollama |
| RTX Desktop (24 GB VRAM) | Gemma 4 12B | Gemma 4 26B-A4B / 31B | LM Studio or llama.cpp |
Phase 2: Run a minimal 4K smoke test and verify acceleration
After installing your chosen runtime and pulling Q4_K_M weights, test a short single-turn text prompt at 4,096 context tokens. Do not start by pasting a 30,000-token codebase or enabling experimental tool-calling plugins.
Immediately verify that hardware acceleration is actually executing the forward pass. Look for 100% GPU residency in `ollama ps`, Dedicated VRAM climbing cleanly in Windows Task Manager, or GPU History spikes in macOS Activity Monitor.
- If your first prompt outputs under 3 tokens per second on a desktop GPU, stop and fix PCIe offloading or missing CUDA/Vulkan backends before testing anything larger.
- If load hangs or exits immediately on mobile, close background apps and step down from E4B to E2B.
- Once short text replies stream smoothly, enable Flash Attention and Q8_0 KV cache quantization before pushing context to 16K–32K.
Phase 3: Unlock multimodal vision, coding tools, and APIs
Gemma 4 checkpoints (especially E4B, 12B IT, and 26B-A4B IT) natively support image inputs, structured reasoning, and agent tool calls—provided your runtime supplies the current Gemma 4 Jinja chat template.
Once your baseline chat feels effortless, you can wire Ollama (`http://localhost:11434/v1`) or LM Studio’s local server (`http://localhost:1234/v1`) into coding assistants such as Continue, Roo Code, Open WebUI, or local Python scripts.
Treat tool calling and strict JSON output as your final graduation test: if a Q4_K_M checkpoint occasionally drops a JSON bracket on long schema prompts, keep the same model size and step quantization up to Q6_K or Q8_0 before jumping to a heavier parameter count.
# 1. Start with a fast baseline pull in Ollama
ollama run gemma4:e4b
# 2. Confirm 100% GPU offload in a second shell
ollama ps
# 3. Call the local OpenAI-compatible chat endpoint
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"gemma4:e4b","messages":[{"role":"user","content":"Summarize local KV cache tradeoffs in 3 bullets."}]}'Go from zero to working local Gemma 4 in 5 steps
- Run the hardware compatibility checker on the homepage to identify your device memory pool.
- Pick the smallest recommended Gemma 4 checkpoint for your tier (E2B on phones, E4B or 12B on 16 GB desktops).
- Install LM Studio (GUI) or Ollama (CLI/API) on desktop, or AI Edge Gallery on Android.
- Download Q4_K_M weights and run one short prompt at 4K–8K context.
- Verify 100% GPU or Metal acceleration before expanding context length or enabling tool calling.
Recommended models
Frequently asked questions
- Which Gemma 4 file format should beginners download first?
- Start with a GGUF build at Q4_K_M quantization (or native LiteRT inside AI Edge Gallery on Android). Q4_K_M balances strong answer quality with ~70% smaller memory needs than FP16.
- Should I use Base or IT (Instruction-Tuned) Gemma 4 checkpoints?
- Always pick IT (instruction-tuned) checkpoints such as gemma4-e4b-it or gemma4-26b-a4b-it for chat, coding, vision, and assistant workflows. Base checkpoints are intended for custom fine-tuning and raw text completion.
- How can I verify my machine is actually offline during inference?
- Once model weights finish downloading in LM Studio, Ollama, or PocketPal, disconnect Wi-Fi or block outbound access—every prompt and generation runs 100% locally on your CPU/GPU.
- What if Gemma 4 feels slower than an online browser chat?
- Browser playgrounds run on datacenter GPUs. If your local speed sits below 10 tokens/sec, check whether layers spilled out of VRAM or step down one model size (for example 26B to 12B or E4B).
- Do I need Python or coding knowledge to run Gemma 4 locally?
- No. GUI apps like LM Studio on Mac/Windows and AI Edge Gallery or PocketPal on mobile let you search, download, and chat with zero command-line setup.