By Sherry Xu, Prashant Ranjan, Torsten Hoefler
As we enter the era of frontier-scale intelligence, the economics of AI are increasingly determined not by peak compute performance but by how efficiently an entire system can generate tokens at scale. Production inference workloads are rapidly expanding beyond interactive chat toward an array of Copilots, coding assistants, deep reasoning, multimodal experiences, agentic workloads, and synthetic data generation. Azure Maia 200, Microsoft’s second-generation AI accelerator, is our latest custom silicon, engineered from the ground up for efficient inference at cloud scale.
At approximately 30% better performance per dollar than the latest-generation GPUs in Microsoft’s fleet, Maia 200 is built on one central principle: that co-optimizing the models, application harnesses, reinforcement learning environments, kernels, communication library, systems, and custom silicon will lead to the best customer outcomes – per token, per watt, and per dollar.
There is no single specification that can achieve this, and no equation that drives faster math. System-level efficiency emerges from co-optimizing models, software, data placement, communication, and silicon together. At Hot Chips 2026, we are pleased to share the foundation behind that effort: the Software-defined Local Access architecture embodied in Maia 200.
Inference is becoming a systems challenge
Inference is no longer a compute problem alone—it is a systems problem that spans memory, networking, and software. As the range of production inference use cases continues to expand, workloads have also become more diverse and demanding.
Different phases of inference place different stresses on infrastructure: processing an initial prompt often requires substantial compute, while generating tokens over time can become constrained by memory bandwidth and communication. Emerging model architectures, including Mixture-of-Experts designs, sparse execution techniques, and advanced KV-cache strategies, further increase the importance of efficiently moving and managing data. At the same time, the largest models now span many accelerators, making communication between devices a critical factor in overall performance.
Meeting these demands requires optimizing for more than peak operations per second. An effective inference platform must deliver high throughput, low latency, reliability, and model quality while minimizing cost and energy per token. Maia 200 was designed with this broader objective in mind, and approaches this challenge using the architectural principles of Software-defined Local Access—both inside the accelerator and scaling to an all-Ethernet architecture that extends explicit memory orchestration across the system.
Software-defined Local Access: making movement explicit
Software-defined Local Access, or SDLA, is a dataflow architecture that gives software direct control over how data moves between high-bandwidth memory and localized, highly specialized SRAMs. Rather than relying on implicit cache behavior, SDLA separates control from data movement and gives software direct control over both memory movement and placement using Direct Memory Access (DMA) engines, compute engines, network operations, and synchronization. This approach helps the system achieve more predictable execution characteristics, enabling more consistent performance between runs.
Maia 200 brings the SDLA principles together in a vertically co-designed system-on-chip. Its control, compute, memory, I/O, and synchronization structures are designed to operate concurrently rather than compete for a single control path. The purpose of the architecture is to keep the compute engines productively fed by placing, reshaping, moving, and synchronizing data efficiently.
All-Ethernet networking: extending SDLA beyond the chip
As models span accelerators, networking becomes part of the execution engine. Maia 200’s two-tier scale-up network applies an all-Ethernet approach, combining a standards-based physical and switching foundation with AI-specific endpoint hardware, transport, topology, congestion management, reliability mechanisms, and collective software.
Maia 200 delivers scalable, predictable performance for large AI inference deployments through an advanced two-tier, Ethernet-based scale-up networking architecture. Intelligent load balancing distributes traffic across on-chip resources, while built-in resilience mechanisms quickly adapt to network disruptions. This enables customers to run dense inference workloads more efficiently, maximize cluster utilization, reduce stranded capacity, and lower overall infrastructure costs.
We have contributed this direction to the Ultra Ethernet Consortium (UEC)’s AI base transport profile, helping establish Ethernet as a common foundation for networking across the industry.
Figure 1. Maia 200 uses a unified Ethernet hierarchy based on a variant of the HammingMesh topology: four accelerators form a directly connected quad, 48 accelerators form a rack-scale domain, and a two-tier switched network can scale to 6,144 accelerators.
Sustaining performance across real AI workloads
The value of the SDLA architecture appears in sustained kernel performance rather than peak specifications alone. To evaluate Maia 200 under realistic conditions, we measured performance across 6,143 matrix-multiplication shapes representative of production inference workloads. Across these scenarios, Maia 200 kept compute engines highly utilized, minimizing idle time by coordinating computation, memory, and communication as a single workload. Even as AI workloads shift between heavy computation (e.g., prompt processing), memory access (e.g., token generation), and communication (e.g., collective operations such as Allgather), Maia 200 maintains consistently predictable performance—treating the accelerator, memory, network, and software as one coordinated system.
Figure 2. FP8 matrix-multiplication performance across the same set of inference-relevant shapes, showing high utilization across compute- and memory-bound operating points.
Figure 3. Allgather performance on eight Maia 200 accelerators. Direct exchange minimizes latency for small transfers; ring exchange approaches the network bandwidth limit for larger transfers.
From silicon to useful AI systems
Maia 200 demonstrates Microsoft’s system co-design philosophy for model performance and efficiency. Maia 200 delivers over 40% higher token generation under the same rack power budget throughput when running the MAI-Thinking-1 model than other leading accelerators in the Azure fleet. This efficiency is only possible by co-designing models, application frameworks, reinforcement-learning environments, kernels, communication libraries, systems software, and custom silicon as a unified stack. At the same time, Maia 200 maintains the flexibility to support a broad ecosystem of open models, enabling customers to optimize for both performance and choice.
As Microsoft continues along a path of co-designing models and infrastructure, Maia 200 provides a foundation for improving quality, performance, cost, and energy efficiency for agentic inference workloads. We look forward to advancing these SDLA architectural principles in future generations of Maia systems.
To learn more, explore our blogs on Maia 200: