[CLSTR]
< Back to situation

[REVISION HISTORY]

Strata engine for local large language model inference

Updated 1 time since CLSTR started tracking revisions of this situation.

What changed

2026-10-08 02:37 UTC → 2026-10-26 01:21 UTC · added removed

The open-source Strata inference engine, developed by Niko1221, has been released Niko1221 (also referred to enable as Niko Veit), enables the local execution of massive large language models on consumer-grade hardware. The engine allows users to run the 125-billion-parameter Qwen3.8-Flash-Next model manages memory constraints by distributing the workload workloads across a machine’s GPU, CPU, RAM, and SSD. To manage memory constraints, Strata It utilizes expert-level three-tier caching and caching—loading high-frequency experts into VRAM while keeping others in RAM or SSD—and speculative decoding. This approach enables hardware decoding to accelerate inference. Initially, Strata enabled the 125-billion-parameter Qwen3.8-Flash-Next model (a Mixture-of-Experts model released by Alibaba in August 2026) to run on systems with at least 12 GB of VRAM and 32 GB of RAM to perform inference. RAM. Performance testing shows significant speeds, such as showed speeds of 94 tokens per second on an NVIDIA GeForce RTX 5070 using a 2-bit quantized version, 53 tokens per second with a 3-bit version, and up to 120 60 tokens per second on a an AMD Radeon 7900 XTX. RX 9070 XT using 2-bit quantization. Strata has since expanded to support the 763-billion-parameter DeepSeek-V4.1-Flash model. By utilizing the ‘GSQ-RCO’ quantization method developed by the ISTA-DASLab at the Austrian Institute of Technology, the engine allows this model to operate on consumer PCs. A system with a 12GB GPU and 46GB of RAM can achieve speeds of 8 to 10 tokens per second. Available quantized versions range from 1.75-bit to 2.25-bit precision; the 1.75-bit version reportedly maintains ‘98.7% of the original model’s average performance’ across mathematics, knowledge QA, and programming benchmarks. Designed for privacy and local operation, privacy, the engine eliminates cloud costs and ensures data remains on the user’s machine. Recent updates to the software, security updates, such as version 0.1.38, have included security include patches to block DNS-rebinding and cross-site attacks.

Versions

  1. 2026-10-26 01:21 UTC Strata engine for local large language model inference
  2. 2026-10-08 02:37 UTC Strata engine for local large language model inference

Only revisions since CLSTR began indexing content versions appear here. Select a version to see what changed compared to the one before it.