Meta provided a detailed glimpse into its next-generation AI infrastructure, with multiterabit networking and open systems set to power workloads for its Muse agents.

At this week's OCP event in San Jose, California, Nvidia already revealed to the world Meta’s IceCubes custom scale-out network switch built on Spectrum-6. But the hyperscaler is hedging its bets when it comes to application-specific integrated circuit (ASIC) switches, revealing the switches would also use Broadcom’s Tomahawk 6.

Both sets of chips provide 102.4Tbps of bandwidth – a twofold increase on previous platforms. They’ll run Meta’s open-source Facebook Open Switching System (FBOSS) network operating system to dynamically route packets across a multiplane configuration, distributing GPU traffic across up to eight independent network planes, providing for fewer layers of switches and therefore improved resiliency should a system fail.

Beyond opting for diversity in switching hardware, Meta also provided more details on the network racks set to house them.

IcePacks are based on the OCP’s ORV3-HPRv1 standard and are the hyperscaler’s first liquid-cooled network racks. Each blade supports 128 separate 800Gbps (800G) ports operating simultaneously, and its non-blocking design allows every single port to run at maximum capacity without causing bottlenecks under heavy AI loads, ensuring GPU utilization.

IceCube scale-out switches were joined by the less chilly-sounding Tahan scale-up switch.

Tahan is built on Broadcom’s Tomahawk 5 ASIC and provides 51.2Tbps of total bandwidth. Unlike its larger scale-out cousin, the Tahan switch is highly condensed by design, with four switches interconnected to 16 of Meta’s MTIA 300 custom silicon chips to provide 6.4Tbps per accelerator.

The switch also reuses the same remote direct memory access (RDMA) over converged Ethernet (RoCE) transport as Meta’s scale-out network to form a unified fabric across both scale-up and scale-out that’s united via FBOSS, keeping the entire fabric neatly tied together.

Meta also provided a peek at its yet-unnamed next-generation scale-up blade. This blade is set to be built on Broadcom’s Tomahawk 6 chips – eight of which can be interconnected to 72 accelerators per rack, marking a sizable increase over Tahan. Besides boosting density, the 102.4T-gen blade will also support Ethernet for Scale-Up Networking (ESUN). No timeframe for when this next-generation scale-up blade will debut was given.

Fabric multiplane makeover & scale-across aggregation

Meta also revealed a new multiplane backend scale-out fabric.

Dubbed Flattened Planar Fabric (FPF), it replaces traditional single-plane uplinks by connecting each GPU through eight independent 100G links to eight distinct switches, one per network plane. The idea is to improve fault tolerance by allowing the host to steer traffic away from downed planes. The switch from a three-tier to a two-tier fabric, meanwhile, removes the need for more switches, which in turn reduces unnecessary hops and lowers both latency and power.

Meta revealed Nvidia engineers worked closely with the Facebook parent on developing FPF, tapping innovations from Spectrum-X multiplane configurations. The pair also worked with unnamed cloud partners to “help standardize multiplane fabric implementations” based on its learnings with FPF.

“This work with cloud providers on FPF is just one example of how we at Meta are not just building our own infrastructure, but are helping our partners’ infrastructure,” Meta's network infrastructure team noted in a post.

Meta’s OCP unveilings also saw it detail backend aggregation (BAG), a centralized, Ethernet-based “super-spine” for interconnecting multiple disparate fabrics into one congealed mega cluster.

The scale-across concept works for systems spanning buildings, campuses, and even facilities spread across regions, with Meta claiming it allows for capacity between sites to reach tens-of-petabits per second.

BAG’s development marks Meta's development of vast gigawatt-scale facilities, including Prometheus in Ohio and Hyperion in Louisiana. The backend aggregation layer is designed to allow these distributed sites to connect as a single cluster. It uses what Meta network engineers billed as “eBGP (exterior border gateway protocol) link-bandwidth signaling,” which sees traffic distributed in proportion to the capacity actually available on each path, with damping in place to prevent a single link failure resulting in routing churn across the stack.

BAG also encompasses adaptive routing improvements to reduce latency and scheduling aimed at minimizing how much collective traffic needs to cross the layer.

“Training remains resilient throughout. When a replica fails, it is automatically excluded, and the surviving replicas continue synchronizing across BAG,” Meta engineers wrote. “The direction of travel is longer distances still – training jobs spanning regions tens, hundreds, or thousands of kilometers apart – and increasingly heterogeneous clusters that mix hardware generations and vendors.”