API
< Back to all clusters
[TECHNOLOGY] · United States · 5 sources

started · updated

Supermicro, IBM, and NVIDIA launch joint KV cache solution

Supermicro, IBM, and NVIDIA have announced a collaborative KV (key-value) cache storage solution designed to enhance AI inference performance. The integrated architecture combines Supermicro Petascale all-flash storage servers, IBM Storage Scale software, NVIDIA Dynamo inference framework software, and NVIDIA Spectrum-X Ethernet networking.

The solution aims to address bottlenecks in agentic and long-context AI workflows by extending KV cache capacity beyond local GPU memory. This allows for higher GPU utilization, increased throughput, and improved infrastructure scalability. By storing pre-calculated KV states in persistent storage, the system avoids the need to recompute the same context for subsequent requests.

Performance testing using a 120 billion parameter large language model demonstrated that for input lengths of 130,000 tokens, the persistent KV cache reduced the Time-to-First-Token (TTFT) by up to 56 times compared to systems without the cache. The technology is intended to provide a more resilient and scalable inference architecture for enterprises and cloud service providers.

Entities

Charles Liang · IBM · Nvidia · Supermicro