Troubleshooting
Gemma 4 GPU or NPU Not Working
Fix Gemma 4 CPU fallback across NVIDIA CUDA, Apple Metal, AMD ROCm/Vulkan, Intel Arc SYCL, and Android NPU delegates with concrete verification commands.
A classic local AI trap is seeing a model answer slowly while your CPU runs at 100%, your fans scream, and your expensive GPU sits near 0% utilization. Because most runtimes fall back silently to CPU execution when a GPU driver or library mismatches, you must verify hardware acceleration explicitly.
Use the platform-specific diagnostics below to confirm real GPU/NPU execution and resolve missing backend delegates.
How to verify GPU or NPU acceleration per vendor
Do not guess from token speed alone—check hardware telemetry directly during generation:
| Accelerator | Telemetry Verification Check | Common Silent Failure Cause | Direct Fix |
|---|---|---|---|
| NVIDIA CUDA | `nvidia-smi` shows runtime process + VRAM | CUDA runtime mismatch or missing Toolkit DLL | Update NVIDIA Game Ready/Studio driver; switch to Vulkan/CUDA 12 pack |
| Apple Silicon Metal | Activity Monitor -> GPU History climbs | Downloaded x86_64 Intel build running under Rosetta | Install native arm64 Apple Silicon app/Homebrew binary |
| AMD Radeon (Win/Linux) | Task Mgr / `rocm-smi` shows Compute & VRAM | Consumer RX card lacks ROCm target override | Use Vulkan backend in LM Studio/llama.cpp, or set `HSA_OVERRIDE_GFX_VERSION` |
| Intel Arc | Task Mgr Compute / `intel_gpu_top` | Standard binary compiled for CPU AVX2 only | Use llama.cpp SYCL or Vulkan build (`GGML_VULKAN=ON`) |
| Android NPU / GPU | AI Edge Gallery backend log & tok/s > 10 | Unrecognized SoC driver or unsupported tensor op | Toggle GPU OpenCL delegate vs NPU HTP in app settings |
Diagnostic terminal checks for desktop GPUs
Run the commands below while generating a long 300-token reply to confirm layers are offloaded to your accelerator:
# NVIDIA: watch GPU utilization & VRAM allocation live
nvidia-smi -l 1
# Ollama: confirm 100% GPU processor assignment
ollama ps
# llama.cpp: verify startup logs contain Metal / CUDA / Vulkan buffer allocation (not CPU only)
llama-cli -hf ggml-org/gemma-4-E2B-it-GGUF -ngl 99 -p "Hi" -n 32 2>&1 | grep -E "offloaded|CUDA|Metal|Vulkan"Rules for clean multi-platform troubleshooting
- On Apple Silicon Macs, check Activity Monitor -> Kind column: if LM Studio, Ollama, or terminal shows "Intel" instead of "Apple", Metal acceleration is disabled under Rosetta 2.
- On Windows NVIDIA laptops with hybrid Optimus graphics, ensure the runtime uses the High-Performance NVIDIA processor rather than integrated Intel UHD graphics.
- On AMD Radeon RX 6000/7000/9000 GPUs on Windows, Vulkan backend builds inside LM Studio and llama.cpp currently require zero manual ROCm path hacks and deliver reliable acceleration.
Linux GPU permissions, WSL2 CUDA passthrough, and Docker GPU flags
Linux and Windows WSL2 setups introduce two additional failure modes where GPU detection silently fails. On native Linux distributions running AMD ROCm or Vulkan, confirm your user account belongs to both the `video` and `render` system groups (`sudo usermod -aG video,render $USER`) so `/dev/kfd` and `/dev/dri` render nodes are readable by non-root daemons like `ollama`.
Inside Windows WSL2 Ubuntu terminals, never install native Linux NVIDIA GPU drivers on top of the Windows driver—install only the CUDA user-space toolkit and rely on Windows WDDM `/dev/dxg` passthrough, or run native Windows Ollama/LM Studio directly for cleaner VRAM management. Inside Docker containers, always pass `--gpus all` (NVIDIA) or `--device=/dev/kfd --device=/dev/dri` (AMD) or inference silently falls back to CPU.
Finally, confirm your runtime is not compiled against an unsupported GPU compute capability—older Pascal/Turing cards or integrated iGPUs occasionally require explicit Vulkan or older CUDA 12.x binary packs instead of bleeding-edge CUDA 13 builds.
Restore missing GPU acceleration in 4 steps
- Confirm your runtime binary matches your architecture (native arm64 on Mac, CUDA/Vulkan build on Windows/Linux).
- Explicitly pass `-ngl 99` (or set GPU Offload slider above zero in LM Studio) so layers do not remain on CPU.
- Monitor `nvidia-smi`, Activity Monitor GPU History, or `ollama ps` during a prompt to verify active compute.
- If CUDA fails on a cutting-edge driver mismatch, temporarily switch to the Vulkan runtime pack as a clean cross-vendor fallback.
Recommended models
Frequently asked questions
- Why does `llama.cpp` print `offloaded 0/29 layers to GPU`?
- Either `-ngl` was omitted (defaulting to 0 on older builds), you downloaded a CPU-only binary without CUDA/Vulkan/Metal compiled in, or the GPU driver failed initialization.
- Can AMD Radeon GPUs run Gemma 4 smoothly on Windows?
- Yes. Use the Vulkan backend in LM Studio or `llama.cpp` (`-DGGML_VULKAN=ON`), which works reliably across Radeon RX 6000, RX 7000, RX 9000, and Radeon AI PRO cards without complicated ROCm setup.
- Why is Apple Silicon Metal failing inside my Python or CLI terminal?
- If your terminal or Homebrew environment runs under Rosetta (`arch` prints `i386`), builds compile without native arm64 Metal support. Run `arch -arm64` and reinstall native Apple Silicon binaries.
- Does Android NPU support every quantization format?
- No. Mobile NPUs require specific integer layouts packaged in LiteRT `.task` bundles inside AI Edge Gallery. Generic desktop GGUF files run via CPU or OpenCL GPU shaders instead.
- How fast should Gemma 4 E4B run when GPU acceleration IS working?
- On any modern desktop GPU or Apple M-series Mac with working Metal/CUDA/Vulkan, Gemma 4 E4B Q4_K_M typically exceeds 20–60+ tokens per second—if you see 2–4 tok/s, it fell back to CPU.