Orchestro unveils AI inference optimization technology to boost GPU throughput

Orchestro, an AI infrastructure software specialist, has announced new strategies to optimize AI inference performance and GPU utilization. The company revealed performance test results showing that its 'CONCERTO AI' platform can increase AI inference throughput by 62.8% and reduce token generation latency by 52.1% within the same GPU environment.

To address resource contention in Large Language Models (LLMs), CONCERTO AI utilizes distributed inference technology that separates the 'prefill' (input processing) and 'decode' (answer generation) stages. This separation minimizes competition for GPU resources during complex tasks, such as those required by Agentic AI.

Additionally, Orchestro demonstrated infrastructure improvements through integration with NetApp storage technology. By utilizing GPUDirect Storage (GDS) for KV cache offloading, the company reduced the time required to load AI model weights from 71 seconds to 8 seconds. The time needed to deploy new servers into service was also reduced from 187 seconds to 80 seconds.

Orchestro plans to expand its support for various hardware, including NVIDIA GPUs and domestic NPUs, to help enterprises operate independent AI infrastructures. The company will host an online webinar on October 27 to discuss these optimization strategies in detail.