]> Cisco Systems, Inc.

Belgium kmichiel@cisco.com

Cisco Systems, Inc.

Belgium cf@cisco.com

Alibaba Cloud

United States of America eddie.ruan@alibaba-inc.com

Alibaba Cloud

China galadriel.zyq@alibaba-inc.com

OPS SRv6 Operations SRv6 Route Reflector AI AI clusters and hyperscale data centers can contain hundreds of leaf switches and large numbers of service prefixes. In a conventional BGP leaf-spine design, each spine distributes routes for the service prefixes originated by one leaf to all other leaves. This leads to a large number of received BGP route instances on the leaves and many FIB entries for service prefixes on the spines. This document describes an alternative deployment in which dedicated eBGP speakers, referred to in this document as Service RRs, distribute service routes. The spines continue to distribute infrastructure routes but do not carry service routes. A Service RR uses regular eBGP procedures while advertising service routes with NEXT_HOP unchanged. The leaves encapsulate the service traffic and the spines forward the encapsulated service traffic using infrastructure reachability. The design improves control- and data-plane scaling by reducing the number of service-route instances received by each leaf and removing service-route state and processing from the spines. This document defines no new BGP extension.

Introduction AI clusters and large data-center networks commonly use BGP as their routing protocol . A typical design uses a two-tier folded Clos topology in which every leaf has an eBGP session with every spine. The spines share one ASN, and each leaf uses a distinct ASN. The BGP sessions distribute both infrastructure reachability and service reachability. The fabric offers many equal-cost paths between leaves. When service routes are distributed through the spines, every spine advertises a route for every remote service prefix to every leaf. The number of received BGP route instances is multiplied by the number of spines. This document describes a deployment that separates the two route distribution functions. Existing leaf-spine eBGP sessions continue to distribute infrastructure routes. A separate set of eBGP speakers, called Service RRs, distributes the service routes. Each leaf adds a multihop eBGP session to each Service RR. The term Service RR does not indicate use of the iBGP route-reflection procedures defined in . A Service RR is a regular eBGP speaker. It applies the existing BGP procedures defined in , including best-path selection and AS_PATH processing. When advertising a service route to a leaf, its one special behavior is to leave the BGP NEXT_HOP unchanged. The next hop therefore continues to identify the leaf that originated the service route. The receiving leaf resolves that next hop through the infrastructure routes distributed by the spines. If the service route carries an SRv6 Service SID, the Service RR also propagates the associated SRv6 Service TLV, as specified in for advertisements that retain the BGP next hop. The Service RRs do not forward service traffic. An ingress leaf encapsulates the traffic in IPv6. The outer IPv6 destination identifies an encapsulation destination reachable through the infrastructure routes, such as an egress-leaf address or an SRv6 Service SID. The spines forward the encapsulated packet using infrastructure reachability. The benefit of this design is a reduction in service-route instances at each leaf and the removal of service routes and their update processing from the spines. This document does not define new BGP extensions or data-plane behavior.

Terminology The following terms are used in this document: BGP route: A service or infrastructure prefix together with the BGP path attributes associated with that prefix. BGP route instance: A BGP route received from a particular peer. A leaf can receive multiple route instances for the same prefix from different peers. Infrastructure prefix: A prefix used to reach a switch loopback, link, tunnel endpoint, SRv6 locator, or other part of the fabric infrastructure. Service prefix: A prefix used to reach a server, workload, or service endpoint. Service RR: An eBGP speaker that receives service routes from leaves and advertises selected service routes to other leaves. It uses regular eBGP procedures and preserves NEXT_HOP when propagating a service route. It does not use the iBGP route-reflection procedures of .

Existing Leaf-Spine Design

Topology and Addressing Model Consider a full-bandwidth two-tier folded Clos in which each spine has one link to each leaf. This document uses rounded values to make the calculations easier to follow: Assuming one infrastructure prefix per switch, the fabric contains 700 infrastructure prefixes. Assuming one service prefix per endpoint, the fabric contains 100,000 service prefixes, corresponding to 500 leaves with 200 endpoint prefixes per leaf in this example. Following the ASN-allocation style described in , this document assumes that all spines use a common ASN and that every leaf uses a distinct ASN. The common spine ASN prevents a route received from one spine from being advertised by a leaf to another spine, because the receiving spine's ASN would already be present in the AS_PATH. Servers do not need to run BGP in this model. A leaf can learn locally attached service reachability by connected or static routing and originate the corresponding service routes into BGP.

Infrastructure-Route Distribution Every leaf has a direct eBGP session with every spine. Each leaf originates and advertises a route for its infrastructure prefix to every spine. A spine advertises a leaf's infrastructure route to all other leaves. Each spine originates and advertises a route for its own infrastructure prefix to every leaf. For S = 200 and L = 500, there are 700 infrastructure prefixes: 200 spine prefixes and 500 leaf prefixes. Since a spine receives one leaf infrastructure route from each of the L leaves, it holds L received leaf-originated routes, in addition to its locally originated infrastructure route. A leaf receives one directly originated spine route from each of the S spines. It also receives one route from each spine for every other leaf's infrastructure prefix. Its received infrastructure-route count is therefore: For S = 200 and L = 500, a leaf receives 100,000 BGP route instances from its neighbors. These route instances represent 699 remote infrastructure prefixes: 200 spine prefixes and 499 other-leaf prefixes. When supported and enabled, BGP multipath allows the leaf to install multiple ECMP paths for a remote leaf prefix in the forwarding table. The leaf could install the 499 other-leaf infrastructure prefixes with 200 ECMP paths per prefix.

Service-Route Distribution In the existing design, not using Service RRs, each leaf advertises one route for each of its P locally attached service endpoint prefixes to each spine. A spine receives one route for every service prefix, for a total of: With L = 500 and P = 200, each spine receives 100,000 service routes. It advertises each service route to the other leaves. A leaf receives one route from every spine for every service prefix originated by another leaf. Its received remote service-route count is: For the example topology: The leaf receives 19,960,000 remote BGP route instances representing 99,800 remote service prefixes. Together with its 200 locally attached service prefixes, the leaf represents 100,000 unique service prefixes. When supported and enabled, BGP multipath allows the leaf to install multiple ECMP paths for a remote service prefix in the forwarding table. The leaf could install the 99,800 remote service prefixes with 200 ECMP paths per prefix.

Service RR Architecture

Route Separation The Service RR design separates infrastructure-route distribution from service-route distribution:

• The existing leaf-spine eBGP sessions distribute infrastructure routes.
• Leaves do not advertise service routes to the spines, and the spines do not hold or advertise service routes.
• Each leaf establishes an additional multihop eBGP session with each Service RR.
• Leaves advertise service routes to the Service RRs.
• Each Service RR distributes eligible service routes to the leaves.

Import and export policies enforce this separation. Infrastructure routes remain available to resolve BGP next hops and IPv6 encapsulation destinations and to forward encapsulated traffic across the fabric. The Service RR is a control-plane node. It is not in the forwarding path between leaves.

Service RR BGP Behavior A Service RR is a regular eBGP speaker. It performs the existing BGP procedures in . Each Service RR uses an ASN different from every leaf ASN, so every Service RR-to-leaf session is an eBGP session. Service RRs can share an ASN or use different ASNs. This choice can affect multipath eligibility but does not change the route-distribution procedures described here. The behavior specific to this architecture is that the Service RR advertises a service route with NEXT_HOP unchanged. The NEXT_HOP received from the originating leaf is therefore retained when the route is advertised to another leaf. The receiving leaf resolves that next hop through the infrastructure routes learned from the spines. If a service route carries an SRv6 Service SID, the Service RR also propagates the associated SRv6 Service TLV, as specified in for advertisements that retain the BGP next hop. When present, the receiving leaf resolves the Service SID through the infrastructure route that covers its locator. When a route for an IPv4 service prefix is advertised with the originating leaf's IPv6 address as NEXT_HOP, the peers use the Extended Next Hop Encoding capability and procedures specified in . The Service RR preserves that IPv6 next hop when advertising the service route to another leaf. This next-hop encoding applies independently of the encapsulation selected for the service traffic. Service RRs operate independently and do not need to exchange service routes with one another because each leaf advertises its service routes to each Service RR.

Use of Multiple Service RRs Each leaf establishes a multihop eBGP session with each of the Q Service RRs. Every Service RR receives the service routes directly from the originating leaves, independently selects its best routes, and advertises eligible routes to the other leaves. The Service RRs should reside in separate failure domains. No Service RR is a data-plane next hop, so the loss of one Service RR does not directly affect service traffic forwarding, provided that at least one other Service RR remains available and advertises equivalent usable routes. Adding Service RRs increases the number of BGP sessions at a leaf. In the example, a leaf retains its 200 sessions to the spines and adds Q Service RR sessions. With four Service RRs, a leaf has 204 BGP sessions in total.

Scaling Analysis Each Service RR receives one route for each of the L * P service prefixes. In the example, each Service RR receives 100,000 service routes. Each Service RR advertises one selected route for a remote service prefix to a leaf. A leaf can therefore receive Q route instances for each remote service prefix. Its received remote service-route count is: With four Service RRs (Q = 4): Compared with the design without Service RRs, the number of remote service-route instances received by each leaf is significantly reduced. In this example, it is reduced from 19,960,000 to 399,200, a factor of 50. The number of unique service prefixes and associated FIB entries for service prefixes at a leaf remains approximately 100,000. The Service RR design reduces replicated BGP route state and update processing at leaves and removes service-route state and update processing from the spines.

Data-Plane Forwarding

Encapsulation and Resolution The spine routing tables do not contain routes for service prefixes. An ingress leaf therefore encapsulates a service packet in an outer IPv6 packet before sending it across the fabric. The spines forward the outer IPv6 packet using infrastructure routes. The encapsulation determines how the outer destination is selected and resolved. This document describes IP-in-IPv6 and SRv6 encapsulation as examples. For IP-in-IPv6 encapsulation , the BGP NEXT_HOP of the service route identifies the egress leaf rather than the Service RR. The ingress leaf recursively resolves this next hop through an infrastructure route to the egress leaf. That infrastructure route can provide equal-cost paths through multiple spines. The ingress leaf places the original IPv4 or IPv6 packet in an outer IPv6 packet whose destination is an IPv6 address assigned to the egress leaf, such as its loopback address. The egress leaf removes the outer IPv6 header and forwards the inner packet to the service endpoint. For SRv6 encapsulation, an egress leaf allocates an SRv6 Service SID for the applicable forwarding context and associates that SID with the advertised service routes. For a global IPv4 and IPv6 service, the Service SID can be an End.DT46 SID associated with the main IPv4 and IPv6 forwarding tables, as specified in . The egress leaf advertises the service route and its Service SID using the SRv6 L3 Service TLV of the BGP Prefix-SID attribute, as specified in . The Service SID is allocated from an SRv6 locator for which the egress leaf advertises an infrastructure route. The ingress leaf uses the Service SID as the destination of the outer IPv6 header, and the spines forward the packet using the route for the locator. The BGP NEXT_HOP continues to identify the egress leaf and is used to track reachability to that leaf. At the egress leaf, the Service SID behavior processes the packet. If the Service SID is an End.DT46 SID, the egress leaf decapsulates the packet and performs the IPv4 or IPv6 lookup. A service route remains an individual route regardless of the encapsulation. It is usable only while its BGP next hop and the forwarding information required by the selected encapsulation remain resolvable.

Service RR Failure Handling A Service RR does not appear in the forwarding chain. Loss of a single Service RR therefore does not directly break the data plane. The affected eBGP session fails, routes learned only over that session are withdrawn, and regular BGP route selection chooses among the routes remaining from other Service RRs. Forwarding is expected to continue when at least one other Service RR remains established and advertises an equivalent usable route.

Security Considerations TBD

IANA Considerations This document has no IANA actions.

References Normative References This document defines the model and generic mechanisms for IPv6 encapsulation of Internet packets, such as IPv6 and IPv4. [STANDARDS-TRACK] This document discusses the Border Gateway Protocol (BGP), which is an inter-Autonomous System routing protocol. The primary function of a BGP speaking system is to exchange network reachability information with other BGP systems. This network reachability information includes information on the list of Autonomous Systems (ASes) that reachability information traverses. This information is sufficient for constructing a graph of AS connectivity for this reachability from which routing loops may be pruned, and, at the AS level, some policy decisions may be enforced. BGP-4 provides a set of mechanisms for supporting Classless Inter-Domain Routing (CIDR). These mechanisms include support for advertising a set of destinations as an IP prefix, and eliminating the concept of network "class" within BGP. BGP-4 also introduces mechanisms that allow aggregation of routes, including aggregation of AS paths. This document obsoletes RFC 1771. [STANDARDS-TRACK] Multiprotocol BGP (MP-BGP) specifies that the set of usable next-hop address families is determined by the Address Family Identifier (AFI) and the Subsequent Address Family Identifier (SAFI). The AFI/SAFI definitions for the IPv4 address family only have provisions for advertising a next-hop address that belongs to the IPv4 protocol when advertising IPv4 Network Layer Reachability Information (NLRI) or VPN-IPv4 NLRI. This document specifies the extensions necessary to allow the advertising of IPv4 NLRI or VPN-IPv4 NLRI with a next-hop address that belongs to the IPv6 protocol. This comprises an extension of the AFI/SAFI definitions to allow the address of the next hop for IPv4 NLRI or VPN-IPv4 NLRI to also belong to the IPv6 protocol, the encoding of the next hop to determine which of the protocols the address actually belongs to, and a BGP Capability allowing MP-BGP peers to dynamically discover whether they can exchange IPv4 NLRI and VPN-IPv4 NLRI with an IPv6 next hop. This document obsoletes RFC 5549. The Segment Routing over IPv6 (SRv6) Network Programming framework enables a network operator or an application to specify a packet processing program by encoding a sequence of instructions in the IPv6 packet header. Each instruction is implemented on one or several nodes in the network and identified by an SRv6 Segment Identifier in the packet. This document defines the SRv6 Network Programming concept and specifies the base set of SRv6 behaviors that enables the creation of interoperable overlays with underlay optimization. This document defines procedures and messages for SRv6-based BGP services, including Layer 3 Virtual Private Network (L3VPN), Ethernet VPN (EVPN), and Internet services. It builds on "BGP/MPLS IP Virtual Private Networks (VPNs)" (RFC 4364) and "BGP MPLS-Based Ethernet VPN" (RFC 7432). Informative References The Border Gateway Protocol (BGP) is an inter-autonomous system routing protocol designed for TCP/IP internets. Typically, all BGP speakers within a single AS must be fully meshed so that any external routing information must be re-distributed to all other routers within that Autonomous System (AS). This represents a serious scaling problem that has been well documented with several alternatives proposed. This document describes the use and design of a method known as "route reflection" to alleviate the need for "full mesh" Internal BGP (IBGP). This document obsoletes RFC 2796 and RFC 1966. [STANDARDS-TRACK] Some network operators build and operate data centers that support over one hundred thousand servers. In this document, such data centers are referred to as "large-scale" to differentiate them from smaller infrastructures. Environments of this scale have a unique set of network requirements with an emphasis on operational simplicity and network stability. This document summarizes operational experience in designing and operating large-scale data centers using BGP as the only routing protocol. The intent is to report on a proven and stable routing design that could be leveraged by others in the industry.

Acknowledgments The authors thank Ahmed Abdelsalam for his input and review.