BGP PORT EC for AIDC

Introduction With the rapid development of Artificial Intelligence (AI) and Machine Learning (ML), AI workloads often generate large amounts of traffic in data center networks. If the available link bandwidth is insufficient, congestion may lead to packet loss or excessive latency, significantly affecting the completion time and computational efficiency of AI tasks. Within a Point of Delivery (PoD), Leaf switches connect to NICs/GPUs, while Spine switches provide connectivity between Leaf switches. Although multiple equal-cost paths may be available within the network, traffic from multiple sources can converge on the same server-facing port of a destination Leaf switch, causing last-hop congestion. Identifying this potential congestion point requires the address of the destination Leaf switch and the corresponding server-facing port ID. In multi-PoD deployments, Super-Spine (SSpine) switches connect a PoD to Core switches, which provide connectivity to other PoDs. Traffic destined for another PoD may traverse different SSpine switches and their Core-facing ports. Therefore, a Leaf switch may need to learn the addresses of the relevant SSpine switches and their associated port IDs. In deployments without an SSpine layer, the Spine switches directly connected to the Core serve the same purpose. A destination route may be associated with multiple ports on the same switch or with ports on different switches. The switch addresses and port IDs can be used to identify the corresponding server-facing or Core-facing links for bandwidth negotiation before sending large amounts of traffic. For example, a sender or its connected Leaf can use this information to identify the target switch and the port for a bandwidth request. Such negotiation can help coordinate traffic transmission according to available bandwidth, thereby reducing the risk of congestion, packet loss, and excessive latency. BGP is commonly used to distribute destination reachability information in data centers supporting AI and machine learning workloads . However, a destination route does not by itself identify the server-facing port on the destination Leaf or the relevant Core-facing ports on SSpine switches. Advertising switch addresses and port IDs along with destination routes allows receiving devices to learn the association between them. This document introduces the BGP Route Port ID Extended Community (PORT EC) and Aggregated PORT EC for advertising the association between destination routes, switch addresses and port IDs. It describes the advertisement of server-facing Leaf ports within a Point of Delivery (PoD) and the aggregation of Core-facing port information for inter-PoD routes.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in .

Format The Route Port ID Extended Community is abbreviated as PORT EC. The Aggregated Route Port ID Extended Community is abbreviated as Aggregated PORT EC (AGG PORT EC).

PORT EC format

IPv4 PORT EC format Transitive IPv4-Address-Specific Extended Community defined in and with new sub-type "Route Port ID" is used for carry the IPv4 address of a switch and the related port ID to the destination.

IPv4 Route Port ID Extended Community Format Figure 1 shows the format of IPv4-Address-Specific Extended Community, where:

Sub-Type: TBD. This indicates that this is the Route Port ID extended community;
Global Administrator: 4 octets, set to the IPv4 address of the switch that advertises the route. This address can be the loopback address for establishing the BGP connection;
Local Administrator: 2 octets, set as the port ID of the switch leading to this route, with a value range of 1 to 65535.

IPv6 PORT EC format Transitive IPv6-Address-Specific Extended Community defined in with new sub-type "Route Port ID" is used for carry the IPv6 address of the switch and the related port ID to the destination.

IPv6 Route Port ID Extended Community Format Figure 2 shows the format of IPv6-Address-Specific Extended Community, where:

Sub-Type: TBD. This indicates that this is the Route Port ID extended community;
Global Administrator: 16 octets, set to the IPv6 address of the switch that advertises the route. This address can be the loopback address for establishing the BGP connection;
Local Administrator: 2 octets, set as the port ID of the switch leading to this route, with a value range of 1 to 65535.

AGG PORT EC format The advertisement of Aggregated PORT EC attributes primarily encompasses advertisement of different ports on the same switch as well as advertisement of different ports across different switches. The aggregation procedure and scenario are described in . For multiple ports on the same switch, the Global Administrator values are identical and the Local Administrator values identify different ports. For ports on different switches, each PORT EC carries the corresponding switch address and the port ID.

Specification The following single-PoD and multi-PoD examples explain the PORT EC advertisement procedures in data center networks.

PORT EC advertisement shows an example single-PoD topology. Leaf switches connect to NICs and distribute destination routes through Spine switches. Labels N1, N2, and so on denote NICs and only a subset is shown.

Single-PoD deployment topology Before sending traffic, each Leaf switch, in addition to announcing its own loopback route, also announces routes to connected GPUs/NICs, carrying the extended community attributes defined in this document. The Global Administrator in the PORT EC is set to the loopback address of the Leaf switch, and the Local Administrator in the PORT EC is set to the port ID connected to the GPU/NIC. Routes are forwarded through Spine switches, allowing each Leaf to learn routes to other GPUs within its PoD, including connected Leaf switches and their ports. Upon receiving the route carrying the PORT EC, the leaf switch checks if the address carried in the PORT EC is reachable. If unreachable, the extended community is ignored. If reachable, the address and port information are stored locally or sent to the server. This storing or sending process is outside the scope of this draft. For example, when GPU1 wants to send traffic to GPU20, it needs to ensure that the port between Leaf3 and NIC20 has sufficient bandwidth to receive the traffic. Leaf3 advertises the route to GPU20/NIC20 with a PORT EC containing the address of Leaf3 and the ID of its NIC20-facing port. After propagation through a Spine, Leaf1 learns this association for the destination route.

AGG PORT EC advertisement shows an example multi-PoD topology. Core switches are used to interconnect SSpine switches of different PoDs. SSpine switches connect a PoD to Core switches, which provide connectivity to other PoDs. Only some links are shown in the diagram. In some deployment scenarios, the SSpine layer device may be omitted, and Spine switches directly interconnect with Core switches.

Multi-PoD deployment topology There may be multiple ECMP paths from Spine switches to SSpine switches. When the same advertised route may have multiple different PORT EC attributes, only one route will be selected as the optimal route according to the EBGP forwarding rules within the data center, and then forwarded to other nodes. This does not make good use of resources. For example, Spine1 selects the optimal route received from SSpine switches to other PoDs. Not only SSpine1 can be used as the next hop, but other SSpine switch can also be used as ECMP next hops. However, after Spine switch select the best route with the same prefix, they may only advertise one corresponding PORT EC attribute to Leaf swtiches. In this case, Leaf swithes can only negotiate with the link between Spine1 and SSpine1, and cannot use the ECMP link between Spine1 and other SSpine switches, which will result in a waste of resources. To avoid attribute loss after BGP route selection, one approach is to enable the ADD-PATH function defined in in the PoD. However, this function may cause a route advertisement storm, severely impacting the efficiency of route advertisement and affecting normal forwarding traffic. Another approach is to use the AGG PORT EC attributes defined in this document. This helps establish more comprehensive ECMP entries. When a Leaf switch sends traffic to another PoD, traffic from multiple local sources may converge on the Core-facing ports of the SSpine switches. A PoD may have multiple SSpine switches, each connected to multiple Core switches. To identify these potential congestion points for bandwidth negotiation before traffic transmission, the Leaf needs to learn the relevant SSpine addresses and their Core-facing port IDs. AGG PORT EC provides the aggregated switch and port information associated with the destination route. An SSpine can learn the same destination route from other PoDs through multiple Core-facing ports. When advertising that route toward the local Spine layer, the SSpine uses AGG PORT EC to advertise its address and the corresponding Core-facing port information. This allows the receiving Spine to learn the ports on that SSpine associated with the destination route. When a Spine advertises the destination route to local Leaf switches, it further aggregates the AGG PORT EC information associated with eligible paths to that destination through the SSpine layer. The resulting AGG PORT EC retains the relevant SSpine addresses and their associated Core-facing port information. For example, SSpine1 learns the route to GPU5000 through Core1 and Core2. When advertising the route to Spine1, SSpine1 uses AGG PORT EC to carry its address and the port information for its links to Core1 and Core2. If Spine1 also learns eligible paths through SSpine2 to SSpine8, it further aggregates the AGG PORT EC information received from these SSpines before advertising the route to Leaf1. Leaf1 can then learn the relevant SSpine switches and their Core-facing ports associated with the GPU5000 route.

IANA Considerations IANA is requested to allocate two new code points from the "Transitive IPv4-Address-Specific Extended Community Sub-Types" and the "Transitive IPv6-Address-Specific Extended Community Sub-Types" registry. TABLE_1

Type	Description	Reference
TBD	Route Port ID	This Document

Security Considerations This extension to BGP has similar security implications as BGP Extended Communities , and .