started · updated
Supermicro, IBM, and NVIDIA launch joint KV cache solution
Supermicro, IBM, and NVIDIA have announced a collaborative KV (key-value) cache storage solution designed to enhance AI inference performance. The integrated architecture combines Supermicro Petascale all-flash storage servers, IBM Storage Scale software, NVIDIA Dynamo inference framework software, and NVIDIA Spectrum-X Ethernet networking.
The solution aims to address bottlenecks in agentic and long-context AI workflows by extending KV cache capacity beyond local GPU memory. This allows for higher GPU utilization, increased throughput, and improved infrastructure scalability. By storing pre-calculated KV states in persistent storage, the system avoids the need to recompute the same context for subsequent requests.
Performance testing using a 120 billion parameter large language model demonstrated that for input lengths of 130,000 tokens, the persistent KV cache reduced the Time-to-First-Token (TTFT) by up to 56 times compared to systems without the cache. The technology is intended to provide a more resilient and scalable inference architecture for enterprises and cloud service providers.
Entities
Charles Liang · IBM · Nvidia · Supermicro