Strata engine for local large language model inference

The open-source Strata inference engine, developed by Niko1221 (also referred to as Niko Veit), enables the local execution of massive large language models on consumer-grade hardware. The engine manages memory constraints by distributing workloads across a machine’s GPU, CPU, RAM, and SSD. It utilizes expert-level three-tier caching—loading high-frequency experts into VRAM while keeping others in RAM or SSD—and speculative decoding to accelerate inference.

Initially, Strata enabled the 125-billion-parameter Qwen3.8-Flash-Next model (a Mixture-of-Experts model released by Alibaba in August 2026) to run on systems with at least 12 GB of VRAM and 32 GB of RAM. Performance testing showed speeds of 94 tokens per second on an NVIDIA GeForce RTX 5070 using a 2-bit quantized version, 53 tokens per second with a 3-bit version, and 60 tokens per second on an AMD Radeon RX 9070 XT using 2-bit quantization.

Strata has since expanded to support the 763-billion-parameter DeepSeek-V4.1-Flash model. By utilizing the ‘GSQ-RCO’ quantization method developed by the ISTA-DASLab at the Austrian Institute of Technology, the engine allows this model to operate on consumer PCs. A system with a 12GB GPU and 46GB of RAM can achieve speeds of 8 to 10 tokens per second. Available quantized versions range from 1.75-bit to 2.25-bit precision; the 1.75-bit version reportedly maintains ‘98.7% of the original model’s average performance’ across mathematics, knowledge QA, and programming benchmarks.

Designed for privacy, the engine ensures data remains on the user’s machine. Recent security updates, such as version 0.1.38, include patches to block DNS-rebinding and cross-site attacks.