started · updated
Microsoft unveils Maia 300 AI accelerator platform
Microsoft has detailed its next-generation in-house AI acceleration platform, the Maia 300, designed specifically for inference tasks rather than training. The architecture focuses on long-context, memory-intensive, and decoding-heavy workloads. Each Maia 300 accelerator features eight HBM4 stacks, providing 288 GB of memory and 18 TB/s of bandwidth, with a thermal design power (TDP) of 950W.
The system is designed for massive scale, with 96 accelerators per rack and the ability to interconnect up to 12,288 accelerators into a Superpod using a custom Ethernet-based AI network fabric. This fabric allows for direct peer-to-peer communication between nodes without requiring a host or separate network cards. The Maia 300 utilizes a NUMA architecture to improve data locality and efficiency.
While Microsoft claims the Maia 300 will provide 40% more tokens processed per megawatt compared to leading GPU systems, industry experts have raised questions regarding the metric. Critics argue that “tokens per watt” is an ambiguous measurement that lacks a standardized definition and may inadvertently reward longer, less efficient outputs.
Entities
Azure · GitHub Copilot · Maia 300 · Microsoft · TSMC