RTGWG Working Group W. Cheng Internet-Draft China Mobile Intended status: Standards Track C. Lin Expires: 6 May 2027 New H3C Technologies X. Duan R. Zhuang China Mobile 2 November 2026 Enhanced ECMP for AI Cluster draft-cheng-rtgwg-enhanced-ecmp-03 Abstract In AI training scenarios, per-flow ECMP is the mainstream load balancing technology deployed in current networks. However, inherent hash collision issues of traditional ECMP lead to imbalanced traffic distribution, adversely affecting application performance. To address this problem, this document proposes an enhanced ECMP method that resolves load imbalance caused by hash collisions. The proposed solution effectively improves load balancing efficiency, reduces network congestion, and enhances overall network performance. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 6 May 2027. Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. Cheng, et al. Expires 6 May 2027 [Page 1] Internet-Draft Enhanced ECMP for AI Network November 2026 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 2. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Solution . . . . . . . . . . . . . . . . . . . . . . . . . . 5 3.1. ECMP based on source ingress interface . . . . . . . . . 5 3.2. ECMP based on egress Grouping . . . . . . . . . . . . . . 6 4. Protocol Extension . . . . . . . . . . . . . . . . . . . . . 8 5. Security Considerations . . . . . . . . . . . . . . . . . . . 8 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 8 7.1. Normative References . . . . . . . . . . . . . . . . . . 8 7.2. Informational References . . . . . . . . . . . . . . . . 8 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 9 1. Introduction ECMP (Equal-Cost Multi-Path) is the fundamental load balancing technology widely deployed in AI cluster networks. The dominant implementation of traditional ECMP is per-flow forwarding, as illustrated in Figure 1. Per-flow ECMP distributes traffic across multiple equal-cost paths based on hash calculation of flow characteristics (typically the five-tuple: source IP, destination IP, source port, destination port, protocol). This approach works effectively in general data center scenarios with massive small flows. Its core advantage is that packets of the same flow always follow the same path, naturally guaranteeing in-order packet delivery. However, traditional per-flow ECMP has inherent limitations in load balancing granularity. Its load balancing performance relies on the statistical dispersion of a large number of flows. When the number of flows is limited, or elephant flows account for a high proportion, hash collisions are more likely to occur: multiple large flows are mapped to the same physical link, while other links are underutilized. This results in highly imbalanced link utilization and degraded load balancing performance. Cheng, et al. Expires 6 May 2027 [Page 2] Internet-Draft Enhanced ECMP for AI Network November 2026 In AI training scenarios, per-flow ECMP is still the mainstream load balancing approach. Due to the unique traffic patterns of AI training jobs (i.e., periodic bursts, small number of flows, and elephant flows), the hash collision and load imbalance problems of traditional ECMP become more prominent, which has become a key bottleneck affecting training efficiency. ECMP Flow 1 Hash 1 +------+ Flow 4 +--------- |if 1 |----- Flow 7 +------+ | |----- Flow 2 Hash 2 +------+ Flow 5 +-------- |if 2 |----- Flow 8 +------+ | |----- Flow 3 Hash 3 +------+ Flow 6 +------- |if 3 |----- ... +------+ |... |----- +------+ Figure 1: Per-flow ECMP To address this problem, this document proposes an enhanced ECMP method that resolves load imbalance caused by hash collisions. The proposed solution effectively improves load balancing efficiency, reduces network congestion, and enhances overall network performance. 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 2. Motivation Cheng, et al. Expires 6 May 2027 [Page 3] Internet-Draft Enhanced ECMP for AI Network November 2026 +---------+ +---------+ | Spine1 | | Spine2 | +-#--#-#--+ +#---#--#-+ | | | | | | | | | | | | | | +-----------------------------)-+ | | | | | | | | | | +---------------------------+ | | | | | | | | | | +---)----------+ +------------)-+ | | | | | | | +-#------#+ +-#-----#-+ +--#----#-+ | Leaf1 | | Leaf2 | | Leaf3 | +-#------#+ +-#------#+ +-#------#+ | | | | | | +-#+ +-#+ +-#+ +-#+ +-#+ +-#+ |H1| |H2| |H3| |H4| |H5| |H6| +--+ +--+ +--+ +--+ +--+ +--+ Figure 2: Example of AI training network AI training clusters typically adopt a Spine-Leaf architecture, as shown in Figure 2. Computing nodes are connected to Leaf switches, and multiple parallel equal-cost paths exist between the Leaf and Spine layers. The traffic patterns of AI training networks is significantly different from that of general data centers, making it challenging to achieve balanced load distribution. The core traffic of AI training comes from collective communication operations and parameter synchronization between computing nodes. This traffic pattern has two prominent characteristics that challenge traditional ECMP mechanisms. First, the number of concurrent flows is relatively limited compared with general data centers. Second, AI training flows are typically high-bandwidth, long-duration elephant flows. The statistical load balancing effect of hash-based ECMP cannot be effectively exerted when the number of flows is small. Once hash collision occurs and multiple elephant flows are mapped to the same path, it will cause uneven traffic distribution, potentially leading to network congestion and bandwidth bottleneck, and significantly reducing the overall training throughput. Although packet-based load balancing can mitigate this imbalance to some degree, they introduce packet reordering issues as packets belonging to the same flow may traverse different paths, requiring additional network-level reordering mechanisms. Cheng, et al. Expires 6 May 2027 [Page 4] Internet-Draft Enhanced ECMP for AI Network November 2026 The imbalanced load caused by traditional ECMP has become an important factor restricting the network performance of AI clusters. This document proposes two enhanced ECMP methods to address the load imbalance problem in AI training networks and improve the overall network performance. 3. Solution 3.1. ECMP based on source ingress interface This solution introduces source ingress interface grouping as an additional dimension to participate in ECMP hashing. Group the ingress interfaces for traffic, assign an ECMP number to the interfaces within the same group, and then perform ECMP hashing based on this ECMP number. This method is suitable when the forwarding traffic size for each ingress interface is roughly the same. +-------+ ingress-interface 1 -> group-id 1 -> ECMP Index 1--> |if 1 | +-------+ ingress-interface 2 -> group-id 1 -> ECMP Index 2--> |if 2 | +-------+ ingress-interface 3 -> group-id 1 -> ECMP Index 3--> |if 3 | +-------+ ingress-interface 4 -> group-id 1 -> ECMP Index 4--> |if 4 | +-------+ ingress-interface 5 -> group-id 2 -> ECMP Index 1--> |if 1 | +-------+ ingress-interface 6 -> group-id 2 -> ECMP Index 2--> |if 2 | +-------+ ingress-interface 7 -> group-id 2 -> ECMP Index 3--> |if 3 | +-------+ ingress-interface 8 -> group-id 2 -> ECMP Index 4--> |if 4 | +-------+ Figure 3: ECMP based on source ingress interface As shown in Figure 3, the eight ingress interfaces are divided into two groups, with four interfaces in each group. Within each group: * The ingress interfaces are assigned ECMP numbers 1, 2, 3, and 4 respectively. * ECMP hashing is performed based on these assigned ECMP numbers to select corresponding egress interfaces for forwarding. Cheng, et al. Expires 6 May 2027 [Page 5] Internet-Draft Enhanced ECMP for AI Network November 2026 For traffic entering through the four ingress interfaces in Group 1: Different egress interfaces are selected for forwarding (four distinct paths). Similarly, for traffic entering through Group 2's four ingress interfaces: Different egress interfaces are selected for forwarding (four distinct paths). Compared with traditional per-flow ECMP, this solution reduces the probability that multiple flows are mapped to the same egress path, and effectively improves the uniformity of egress load distribution. 3.2. ECMP based on egress Grouping +-----+ +----+ Egress |Spine|-------|Leaf|----Host Ingress / +-----+ / \ +----+ +----+ / +-----+/ \+----+ Egress Host ----|Leaf|/----|Spine|------|Leaf|----Host +----+\ +-----+\ / +----+ \ +-----+ \ / +----+ Egress \ |Spine|------|Leaf|----Host +-----+ +----+ Figure 4: ECMP based on egress Grouping As shown in figure 4, the source HOST connects to the source Leaf via an Ingress interface, while the destination HOST connects to the destination Leaf through an Egress interface. Multiple ECMP links exist between the Leaf switches and multiple Spine devices. This solution introduces destination egress interface grouping as an additional dimension to participate in ECMP hashing. To improve load-balancing distribution uniformity, the ECMP interfaces connecting to multiple Spine devices are grouped on the source Leaf device. This grouping SHOULD be configured on the Leaf device. For example, if there are 128 equal-cost links between the source Leaf and Spine devices, they can be divided into groups of 4 interfaces each (Group 1: interfaces 1-4; Group 2: interfaces 5-8, etc.). For traffic load balancing, flows MUST first be mapped to specific groups based on their location information (flows with the same location information are assigned to the same group), and then hash- based load balancing is performed within each group. The location information used for group mapping can be either the source's Ingress interface or the destination's Egress interface. Cheng, et al. Expires 6 May 2027 [Page 6] Internet-Draft Enhanced ECMP for AI Network November 2026 Mapping to <------Route(with egress group-id) +-------+Group ID +-----+ +------+ |Ingress|-------------|Spine|-----|Egress|------egress |Leaf | | | |Leaf | +-------+ +-----+ +------+ Figure 5: Route carries remote egress group-id attribute Ecmp Group-- |--Sub Group-id 1 ---------(if-1,if-2, if-3,if-4) |--Sub Group-id 2 ---------(if-5,if-6, if-7,if-8) ... Figure 6: Grouping interfaces within an ECMP Group First, as shown in Figure 5, when the Egress Leaf advertises a route, it MUST carry the egress group index, which can be composed of the local device's router-id and group-id (see Section 4 for details). On the Ingress Leaf, the ECMP egress interfaces toward the Spine devices MUST be grouped (as illustrated in Figure 6). When the Ingress Leaf receives a route, it MUST extract the remote egress group index carried in the route. It then maps this remote egress group index to a local ECMP subgroup index, effectively directing traffic to the corresponding subset of interfaces for forwarding. This ensures that flows destined for different remote addresses are load-balanced across different ECMP subgroups, improving distribution granularity. Refer to Figure 7 for details. +=========+=============+=================+======================+ | Dest |Remote | Local Index | ECMP interfaces | | | Attribute | | | +=========+=============+=================+======================+ | route-1 |Egress Group |Local ECMP |(if-1,if-2, if-3,if-4)| | |Index 1 |Sub Group Index 1| | +=========+=============+=================+======================+ | route-2 |Egress Group |Local ECMP |(if-5,if-6, if-7,if-8)| | |Index 2 |Sub Group Index 2| | +=========+=============+=================+======================+ | ... | ... | ... | ... | +=========+=============+=================+======================+ Figure 7: Mapping of remote egress group index to local ECMP subgroup Cheng, et al. Expires 6 May 2027 [Page 7] Internet-Draft Enhanced ECMP for AI Network November 2026 By implementing fine-grained grouping of ECMP interfaces, this solution achieves more uniform traffic load distribution, thereby addressing the problem of imbalanced load sharing caused by hash collisions. 4. Protocol Extension To support the enhanced ECMP mechanism based on egress grouping described in Section 3.2, the route advertised by the Egress Leaf MUST carry an egress group index, which is composed of the local device's router-id and group-id. The encoding of this information is defined as a new BGP Extended Community (BGP_EXT-COMM-ECMP-GROUPID) in [I-D.cheng-idr-ecmp-groupid]. On the Ingress Leaf, when a route carrying this extended community is received, the remote egress group index is mapped to a local ECMP subgroup index as described in Section 3.2, directing traffic to the corresponding subset of ECMP interfaces for forwarding. 5. Security Considerations TBD. 6. IANA Considerations This document has no IANA actions. The registration request for BGP_EXT-COMM-ECMP-GROUPID is defined in [I-D.cheng-idr-ecmp-groupid]. 7. References 7.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . 7.2. Informational References [I-D.cheng-idr-ecmp-groupid] Cheng, W. and C. Lin, "BGP Extension for ECMP Group ID", Work in Progress, Internet-Draft, draft-cheng-idr-ecmp- groupid-00, 2026, . Cheng, et al. Expires 6 May 2027 [Page 8] Internet-Draft Enhanced ECMP for AI Network November 2026 Authors' Addresses Weiqiang Cheng China Mobile Beijing China Email: chengweiqiang@chinamobile.com Changwang Lin New H3C Technologies Beijing China Email: linchangwang.04414@h3c.com Xiaodong Duan China Mobile Beijing China Email: duanxiaodong@chinamobile.com Rui Zhuang China Mobile Beijing China Email: zhuangruiyjy@chinamobile.com Cheng, et al. Expires 6 May 2027 [Page 9]