Meta’s Distributed AI Infrastructure: Connecting Data Center Fabrics for AI Training
How Meta is connecting its owned, leased, and public cloud infrastructure to support gigawatt-scale AI training.
Hyperscalers assemble AI infrastructure on different timelines. Datacenters can take years to build, power availability can delay deployments, and chip procurement requires advance planning. Here, leased facilities and public cloud capacity can add resources immediately, but they may require connecting infrastructure designed around different systems.
In fact, as per TechInsights’ hyperscale market tracker, at the end of Q2 2026, 60% of operational hyperscale datacenters were in leased facilities, while those facilities accounted for 41% of hyperscale operators’ IT capacity. Based on those shares, the average leased site had less than half the IT capacity of the average self-built site. Leasing, therefore, adds many smaller-capacity blocks rather than a few very large ones.
The challenge is connecting these resources to support large training workloads. At OCP 2026, Meta presented one architectural response.
Why Distributed AI Infrastructure Is Difficult to Unify
Connecting capacity across buildings, campuses, or nearby regions means bringing together infrastructure that may use different GPU vendors or SKUs, network topologies, and fabric technologies. For synchronous training, the slowest or most congested path can limit the workload. A network design that works for a small number of cloud instances may not behave the same way at fleet scale.
Meta’s presentation described an Ethernet superspine in each cluster subregion. It acts as a “backbone of the backbone,” interconnecting many spine-layer fabrics into a single mega cluster spanning buildings, campuses, and regions. Distance still adds latency, with light traveling through fiber taking about 5 microseconds per kilometer one way. That constraint shapes how far the cluster can extend while maintaining the network behavior training workloads require.
Meta uses two main fabric technologies to build its Layer 2 networks—Disaggregated Scheduled Fabric (DSF) and Non-Scheduled Fabric (NSF). The BAG (backend aggregation) layer is like a universal layer that works across these different L2 networks. Inter-BAG connectivity uses a planar design: regions connect one-to-one at the BAG layer, and traffic then fans out across subsets of each region’s Layer 2 fabric. This provides a way to link distinct regional network domains without requiring the underlying L2 fabrics to be identical.
MetaRoCE Spreads Traffic Across Multiple Paths
At the software level, Meta developed MetaRoCE, optimized for AI cluster networks over Ethernet. It is Meta’s version of RoCE (RDMA over Converged Ethernet), a transport protocol that allows one GPU to access another GPU’s memory without involving the CPU.
Standard RoCE typically sends each flow along a single path, which can create hot spots. Meta developed its own version that instead spreads a flow’s traffic across multiple paths. If congestion affects one path, traffic can use other paths, and only the affected data needs to be retransmitted. Meta claims the approach has significantly increased throughput.
Featured Reports
Continue with related TechInsights coverage of server architecture, hyperscaler earnings, and datacenter market data.
- DCC-2609-801Server Architecture Trends - OpenAI’s JalapeñoRead free
- DCC-2608-801Hyperscaler Earnings Quarterly TakeawaysRead free
- DCC-2411-802Market Data: Data Center CAPEX Across US & Chinese hyperscalersView report





