Strata engine enables 125B parameter model inference on consumer GPUs

Developer Niko1221 has released the open-source Strata engine, which enables consumer-grade graphics cards with 12GB of VRAM or more to run the quantized Qwen3.8-Flash-Next model. This 125-billion parameter Mixture-of-Experts (MoE) model was released by Alibaba's Qwen team in August 2026.

Strata utilizes two primary technical methods to manage memory constraints: it implements expert-level three-tier caching—loading high-frequency experts into VRAM while keeping others in RAM or SSD—and employs speculative decoding to accelerate the inference process. In testing on an NVIDIA GeForce RTX 5070, the 2-bit quantized version achieved an inference speed of 94 tokens per second, while the 3-bit version reached 53 tokens per second. An AMD Radeon RX 9070 XT achieved 60 tokens per second using the 2-bit version.

Technical analysis indicates that while Strata provides a general solution for MoE models through memory management, it differs from other specialized approaches like the ds4 project, which is specifically optimized for DeepSeek V4 Flash architectures. Experts suggest that future inference engines will likely converge on a unified paradigm combining heterogeneous memory scheduling, native sparse MoE architectures, and KV cache compression.