Local AI models achieve high performance on consumer hardware

Recent developments in software and quantization techniques are enabling high-performance large language models (LLMs) to run locally on consumer-grade hardware. The release of llama.cpp v0.7.0 introduced a GPU cache for mixture-of-experts (MoE) weights, allowing larger models to operate more efficiently on limited VRAM.

Tests conducted on an Acer Nitro V15 with an Nvidia RTX 4060 (8GB VRAM) demonstrated that the Qwen3.5-35B-A3B model can function as a daily driver, achieving 15 tokens per second. This performance level places it in a similar class to frontier models from the previous year while running entirely locally.

Comparative benchmarks between Qwen3.5-2B and Google’s Gemma 4 E2B on lower-end hardware, such as the RTX 3050 and GTX 1650 Ti, show a trade-off between speed and quality. While Qwen models demonstrated faster decoding and prefill speeds, Gemma 4 E2B maintained higher scoring accuracy across specific tasks when using 4-bit builds.