Moore Threads introduces Prefill-as-a-Service to reduce AI inference costs

Moore Threads has introduced a new technical paradigm called ‘Prefill-as-a-Service’ to address the high costs and latency associated with long-context AI inference. The company released a technical white paper for its flagship MTT S5000 AI training and inference card to explain how this approach decouples the two main stages of large language model processing.

In traditional inference, the ‘prefill’ stage (processing the input) is compute-intensive, while the ‘decode’ stage (generating the response) is memory-bandwidth intensive. Running both on the same hardware often leads to resource inefficiency. Moore Threads proposes separating these tasks into distinct hardware resource pools tailored to their specific requirements.

According to manufacturer specifications, the MTT S5000 can achieve over 4,000 tokens per second during the prefill phase and over 1,000 tokens per second during the decode phase using FP8 precision. This method aims to reduce the total cost of ownership for enterprises managing AI agents, code generation, and extensive document analysis.