Back to blog
Oct 3, 20268 min read

Run Qwen 3.8 Flash 128B on 12GB VRAM: Strata Engine, 96 Tokens/s & Setup Guide

How to run Qwen 3.8 Flash 128B (Qwen3.8-Flash-Next) on consumer 12GB GPUs at up to 96 tokens/s using the Strata engine. Hardware requirements, RAM tiers, and setup.

qwen 3.8 flashqwen 3.8 flash 128bqwen3.8-flash-nextstrata12gb vram

The local AI community and Twitter/X are buzzing over a breakthrough demo: running a 125B/128B open-weights model on an ordinary 12GB gaming PC at nearly 96 tokens per second.

The model is Qwen 3.8 Flash 128B (officially released as Qwen3.8-Flash-Next by Alibaba), and the software making this possible on consumer hardware is Strata (github.com/Niko1221/Strata).

Here is everything you need to know about the model naming, why it fits on a 12GB GPU, real-world benchmark speeds, and how to set it up locally.

Quick answer

  • What model is it? Officially named Qwen3.8-Flash-Next on Hugging Face (125B total parameters, 6B active per token). In community posts and Alibaba API tiers, it is commonly called Qwen 3.8 Flash or Qwen 3.8 Flash 128B.
  • Can a 12GB GPU really run it? Yes. Using the Strata engine, only the active MoE experts (6B) and KV cache are pinned to GPU VRAM, while inactive experts stream from system RAM.
  • What is the real generation speed? On an RTX 5070 (12 GB VRAM), Q2_0 writes at 94 tokens/s (prompt ingest at 2,650 tokens/s); IQ2_XS runs at 79 tokens/s; and the Coder edition runs at 55 tokens/s.
  • What RAM do you need? You need 32 GB RAM for the Coder edition (IQ1_M) or 64 GB RAM to run all quantization sizes (Q2_0, IQ2_XS, IQ3_S).

To inspect exact quantization memory fit against 302 graphics cards, check our Qwen 3.8 Flash 128B model page.

Model naming explained: Qwen 3.8 Flash vs Qwen3.8-Flash-Next

There has been confusion across social media and developer forums regarding the naming of this model:

  1. Qwen3.8-Flash-Next (Open Weights): This is the official open-weights architecture released by Alibaba on Hugging Face (Qwen/Qwen3.8-Flash-Next). It serves as an open architecture preview for the upcoming Qwen 4 family.
  2. Qwen 3.8 Flash (API & Colloquial): This is Alibaba Cloud's hosted production endpoint name. Because developers recognize the "Flash" designation for fast, lightweight frontier models, the community widely refers to the open weights as Qwen 3.8 Flash or Qwen 3.8 Flash 128B.
  3. 125B vs 128B: The model has approximately 125 billion parameters in total (with an additional 51B parameters in off-accelerator n-gram embedding tables). It is often rounded to "128B" in casual discussion.
  4. Strata Engine: The open-source local inference engine built by Niko1221 that packages Qwen3.8-Flash-Next into an automated one-click installer for consumer Windows and Linux PCs.

Why a 125B model runs on 12GB VRAM

Running a traditional 120B+ dense model locally would typically demand at least 70–140 GB of VRAM, requiring multiple enterprise A100 or H100 GPUs. Qwen 3.8 Flash runs on 12GB VRAM through a combination of three architectural innovations:

1. Extreme MoE Sparsity (6B Active)

The model features 512 experts, but routing only activates 8 experts per token—a mere 6 billion parameters active during any single forward pass.

2. Hybrid Attention (Gated DeltaNet + QSA)

Instead of standard quadratic softmax attention across the full 128K context, Qwen 3.8 Flash blends Gated DeltaNet linear attention with Qwen Sparse Attention (QSA). This dramatically compresses KV cache footprint and slashes prefill latency.

3. Tiered VRAM/RAM Memory Streaming in Strata

The Strata engine splits the model:

  • GPU VRAM (12 GB) holds the active execution layers, attention buffers, and active expert working memory.
  • System RAM (32–64 GB) holds inactive experts and the 51B off-accelerator n-gram embedding tables. Because the PCIe bus only needs to stream weights for activated experts, modern high-speed system memory and PCIe 4.0/5.0 bandwidth are sufficient to keep the GPU fed without stalling generation.

Hardware benchmarks: Real tokens per second

Measurements published by the Strata project across consumer gaming hardware:

Setup Model Size Generation Speed Prompt Read Speed RAM Required
RTX 5070 (12 GB) + Ryzen 5 7600 Q2_0 94 tokens/s 2,650 tokens/s 64 GB
RTX 5070 (12 GB) + Ryzen 5 7600 IQ2_XS 79 tokens/s 2,090 tokens/s 64 GB
RTX 5070 (12 GB) + Ryzen 5 7600 Coder (IQ1_M) 55 tokens/s 2,180 tokens/s 32 GB
RTX 3090 (24 GB) Full / Q3 100–140 tokens/s >3,000 tokens/s 64 GB
RX 9070 XT (16 GB) + Ryzen 9 3900X Q2_0 60 tokens/s 1,160 tokens/s 47 GB

Note: 94 tokens per second is substantially faster than human reading speed (which averages 4–5 words or ~6 tokens per second).

How to install and run Qwen 3.8 Flash locally with Strata

Minimum System Requirements

  • Graphics Card: NVIDIA GeForce RTX 20/30/40/50 series or AMD Radeon RX 7000/9000 series with 12 GB VRAM or more.
  • System RAM: 32 GB (for Coder edition) or 64 GB (for full Q2_0 / IQ2_XS editions).
  • Disk: 80 GB of free space on an SSD.
  • Operating System: Windows 10/11 or modern Linux.

Windows Installation Steps

  1. Download the Strata release ZIP from github.com/Niko1221/Strata.
  2. Extract the archive onto your SSD.
  3. Double-click START-HERE.bat.
  4. The automated installer will detect your GPU (NVIDIA or AMD) and recommend the appropriate model quantization based on your installed system RAM.
  5. If prompted, select IQ2_XS (recommended for 64GB RAM) or Coder (for 32GB RAM).
  6. Strata will download the model weights (~70 GB) and start the local server.
  7. Your web browser will open http://127.0.0.1:8080 with a clean web chat interface.

Connecting Coding Assistants & Agents

Strata exposes an OpenAI-compatible and Anthropic-compatible API endpoint directly on http://127.0.0.1:8080/v1. You can point Claude Code, Cursor, Aider, or Roo Code to your local machine for zero-cost, private coding loops.

FAQ

Is Qwen 3.8 Flash the same as Qwen3.8-Flash-Next?

Yes. Qwen3.8-Flash-Next is the open-weights foundation model repository on Hugging Face, while Qwen 3.8 Flash is the hosted service name and community moniker. Both refer to Alibaba's 125B sparse MoE architecture with 6B active parameters.

Can an RTX 3060 12GB run Qwen 3.8 Flash?

Yes! Any modern NVIDIA GPU with 12GB of VRAM (including RTX 3060 12GB, RTX 4070 12GB, and RTX 5070 12GB) meets the VRAM requirement. You will also need 32GB to 64GB of system RAM on the host machine.

Why is generation so fast despite model size?

Because only 6 billion parameters are activated per token, the GPU only executes matrix multiplications for the active experts. Coupled with linear Gated DeltaNet attention and optimized PCIe weight streaming, inference throughput rivals smaller 7B-class models while retaining 125B reasoning depth.

Related posts