vLLM and Ollama performance benchmarks for local and production LLM inference

Recent evaluations of Large Language Model (LLM) inference engines highlight significant performance variations based on hardware and deployment goals. For local inference, the Ollama 0.4 era shows a measurable performance tax compared to bare engines. On an RTX 5060 Ti, Ollama achieved 69.1 tokens per second, approximately 10.3% slower than raw llama.cpp. However, on Apple Silicon, Ollama outperformed LM Studio, delivering 46.2 tokens per second on an M3 Max compared to LM Studio’s 38.2 tokens per second.

For production environments requiring high throughput and concurrency, vLLM remains a primary choice due to its memory-efficient serving capabilities, such as PagedAttention and continuous batching. Integration tools like vllm-haystack allow developers to use vLLM within the Haystack framework to serve open-weights models, such as the Qwen series, via OpenAI-compatible servers.