MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet

MetaRoCE maintains 86% network throughput at a 1% packet loss rate—and keeps working even at 10% loss—running over standard, lossy Ethernet without Priority Flow Control. 👇

Dropping even a fraction of a percent of packets usually causes standard RDMA to stall, stranding massive amounts of GPU compute.

At million-GPU scales, collective operations like all-reduce synchronize thousands of accelerators, meaning the single slowest transfer sets the pace for the entire job. Standard RoCEv2 demands strict in-order packet delivery and relies on pause frames (PFC) to force losslessness. This prevents packet spraying across multiple network paths, causing severe head-of-line blocking and hot spots when links congest or fail.

Meta built MetaRoCE, an open-source RDMA transport protocol designed from scratch for commodity Ethernet. Instead of expecting switches to maintain losslessness and packet order, MetaRoCE moves all transport intelligence to the NIC endpoints and assumes out-of-order arrival and packet loss are normal.

The transport decomposes a connection into fine-grained logical paths and handles data in a continuous, path-aware pipeline:

* Direct-to-memory writes: Every packet carries its final memory destination or receive buffer match. Packets write directly to memory as they land, completely eliminating reorder buffers and head-of-line blocking. * Per-path multipathing: The NIC sprays individual packets across multiple paths using distinct UDP source ports for routing. Each path maintains its own RTT tracking, sequence numbers, and window, letting the NIC route around a broken or slow link without stalling the whole connection. * Targeted loss recovery: A gap in a path's 256-bit selective acknowledgment (SACK) bitvector acts as immediate proof of loss rather than reordering. The endpoint instantly retransmits only the missing packet on that specific path. * Two-sided congestion control: Sender-driven ECN windowing pairs with receiver-driven rate hints returned in every ACK. Senders jump directly to their fair-share speed rather than searching for it, resolving incast congestion in just one or two round trips. * Decoupled connection state: A single connection maps multiple application streams down to multiple network paths under one congestion controller, stopping queue-pair connection state from exploding as cluster parallelism grows.

Meta validated MetaRoCE alongside AMD on a 64-node GPU cluster using Pensando NICs and RCCL collective libraries. MetaRoCE consistently beat RoCEv2 in throughput and latency on all-reduce and all-to-all operations.

While packet loss degraded standard RoCEv2, MetaRoCE retained ~86% throughput under 1% loss and delivered useful bandwidth even at 10% loss. On multiplane topologies with up to 4,000 concurrent connections, throughput scaled linearly with plane count and rerouted traffic automatically during simulated network failures without operator intervention.