Enhanced ECMP for AI Cluster

ECMP (Equal-Cost Multi-Path) is the fundamental load balancing technology widely deployed in AI cluster networks. The dominant implementation of traditional ECMP is per-flow forwarding, as illustrated in Figure 1. Per-flow ECMP distributes traffic across multiple equal-cost paths based on hash calculation of flow characteristics (typically the five-tuple: source IP, destination IP, source port, destination port, protocol). This approach works effectively in general data center scenarios with massive small flows. Its core advantage is that packets of the same flow always follow the same path, naturally guaranteeing in-order packet delivery. However, traditional per-flow ECMP has inherent limitations in load balancing granularity. Its load balancing performance relies on the statistical dispersion of a large number of flows. When the number of flows is limited, or elephant flows account for a high proportion, hash collisions are more likely to occur: multiple large flows are mapped to the same physical link, while other links are underutilized. This results in highly imbalanced link utilization and degraded load balancing performance. In AI training scenarios, per-flow ECMP is still the mainstream load balancing approach. Due to the unique traffic patterns of AI training jobs (i.e., periodic bursts, small number of flows, and elephant flows), the hash collision and load imbalance problems of traditional ECMP become more prominent, which has become a key bottleneck affecting training efficiency.

To address this problem, this document proposes an enhanced ECMP method that resolves load imbalance caused by hash collisions. The proposed solution effectively improves load balancing efficiency, reduces network congestion, and enhances overall network performance.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

AI training clusters typically adopt a Spine-Leaf architecture, as shown in Figure 2. Computing nodes are connected to Leaf switches, and multiple parallel equal-cost paths exist between the Leaf and Spine layers. The traffic patterns of AI training networks is significantly different from that of general data centers, making it challenging to achieve balanced load distribution. The core traffic of AI training comes from collective communication operations and parameter synchronization between computing nodes. This traffic pattern has two prominent characteristics that challenge traditional ECMP mechanisms. First, the number of concurrent flows is relatively limited compared with general data centers. Second, AI training flows are typically high-bandwidth, long-duration elephant flows. The statistical load balancing effect of hash-based ECMP cannot be effectively exerted when the number of flows is small. Once hash collision occurs and multiple elephant flows are mapped to the same path, it will cause uneven traffic distribution, potentially leading to network congestion and bandwidth bottleneck, and significantly reducing the overall training throughput. Although packet-based load balancing can mitigate this imbalance to some degree, they introduce packet reordering issues as packets belonging to the same flow may traverse different paths, requiring additional network-level reordering mechanisms. The imbalanced load caused by traditional ECMP has become an important factor restricting the network performance of AI clusters. This document proposes two enhanced ECMP methods to address the load imbalance problem in AI training networks and improve the overall network performance.

This solution introduces source ingress interface grouping as an additional dimension to participate in ECMP hashing. Group the ingress interfaces for traffic, assign an ECMP number to the interfaces within the same group, and then perform ECMP hashing based on this ECMP number. This method is suitable when the forwarding traffic size for each ingress interface is roughly the same.

group-id 1 -> ECMP Index 1--> |if 1 | +-------+ ingress-interface 2 -> group-id 1 -> ECMP Index 2--> |if 2 | +-------+ ingress-interface 3 -> group-id 1 -> ECMP Index 3--> |if 3 | +-------+ ingress-interface 4 -> group-id 1 -> ECMP Index 4--> |if 4 | +-------+ ingress-interface 5 -> group-id 2 -> ECMP Index 1--> |if 1 | +-------+ ingress-interface 6 -> group-id 2 -> ECMP Index 2--> |if 2 | +-------+ ingress-interface 7 -> group-id 2 -> ECMP Index 3--> |if 3 | +-------+ ingress-interface 8 -> group-id 2 -> ECMP Index 4--> |if 4 | +-------+ ]]> As shown in , the eight ingress interfaces are divided into two groups, with four interfaces in each group. Within each group:

The ingress interfaces are assigned ECMP numbers 1, 2, 3, and 4 respectively.
ECMP hashing is performed based on these assigned ECMP numbers to select corresponding egress interfaces for forwarding.

For traffic entering through the four ingress interfaces in Group 1: Different egress interfaces are selected for forwarding (four distinct paths). Similarly, for traffic entering through Group 2's four ingress interfaces: Different egress interfaces are selected for forwarding (four distinct paths). Compared with traditional per-flow ECMP, this solution reduces the probability that multiple flows are mapped to the same egress path, and effectively improves the uniformity of egress load distribution.

As shown in figure 4, the source HOST connects to the source Leaf via an Ingress interface, while the destination HOST connects to the destination Leaf through an Egress interface. Multiple ECMP links exist between the Leaf switches and multiple Spine devices. This solution introduces destination egress interface grouping as an additional dimension to participate in ECMP hashing. To improve load-balancing distribution uniformity, the ECMP interfaces connecting to multiple Spine devices are grouped on the source Leaf device. This grouping SHOULD be configured on the Leaf device. For example, if there are 128 equal-cost links between the source Leaf and Spine devices, they can be divided into groups of 4 interfaces each (Group 1: interfaces 1-4; Group 2: interfaces 5-8, etc.). For traffic load balancing, flows MUST first be mapped to specific groups based on their location information (flows with the same location information are assigned to the same group), and then hash-based load balancing is performed within each group. The location information used for group mapping can be either the source's Ingress interface or the destination's Egress interface.

First, as shown in , when the Egress Leaf advertises a route, it MUST carry the egress group index, which can be composed of the local device's router-id and group-id (see for details). On the Ingress Leaf, the ECMP egress interfaces toward the Spine devices MUST be grouped (as illustrated in ). When the Ingress Leaf receives a route, it MUST extract the remote egress group index carried in the route. It then maps this remote egress group index to a local ECMP subgroup index, effectively directing traffic to the corresponding subset of interfaces for forwarding. This ensures that flows destined for different remote addresses are load-balanced across different ECMP subgroups, improving distribution granularity. Refer to for details.

By implementing fine-grained grouping of ECMP interfaces, this solution achieves more uniform traffic load distribution, thereby addressing the problem of imbalanced load sharing caused by hash collisions.

To support the enhanced ECMP mechanism based on egress grouping described in , the route advertised by the Egress Leaf MUST carry an egress group index, which is composed of the local device's router-id and group-id. The encoding of this information is defined as a new BGP Extended Community (BGP_EXT-COMM-ECMP-GROUPID) in . On the Ingress Leaf, when a route carrying this extended community is received, the remote egress group index is mapped to a local ECMP subgroup index as described in , directing traffic to the corresponding subset of ECMP interfaces for forwarding.

TBD.

This document has no IANA actions. The registration request for BGP_EXT-COMM-ECMP-GROUPID is defined in .