| Internet-Draft | BGP Service-Route Distribution | November 2026 |
| Michielsen, et al. | Expires 6 May 2027 | [Page] |
AI clusters and hyperscale data centers can contain hundreds of leaf switches and large numbers of service prefixes. In a conventional BGP leaf-spine design, each spine distributes routes for the service prefixes originated by one leaf to all other leaves. This leads to a large number of received BGP route instances on the leaves and many FIB entries for service prefixes on the spines.¶
This document describes an alternative deployment in which dedicated eBGP speakers, referred to in this document as Service RRs, distribute service routes. The spines continue to distribute infrastructure routes but do not carry service routes. A Service RR uses regular eBGP procedures while advertising service routes with NEXT_HOP unchanged. The leaves encapsulate the service traffic and the spines forward the encapsulated service traffic using infrastructure reachability.¶
The design improves control- and data-plane scaling by reducing the number of service-route instances received by each leaf and removing service-route state and processing from the spines. This document defines no new BGP extension.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 6 May 2027.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
AI clusters and large data-center networks commonly use BGP as their routing protocol [RFC7938]. A typical design uses a two-tier folded Clos topology in which every leaf has an eBGP session with every spine. The spines share one ASN, and each leaf uses a distinct ASN. The BGP sessions distribute both infrastructure reachability and service reachability.¶
The fabric offers many equal-cost paths between leaves. When service routes are distributed through the spines, every spine advertises a route for every remote service prefix to every leaf. The number of received BGP route instances is multiplied by the number of spines.¶
This document describes a deployment that separates the two route distribution functions. Existing leaf-spine eBGP sessions continue to distribute infrastructure routes. A separate set of eBGP speakers, called Service RRs, distributes the service routes. Each leaf adds a multihop eBGP session to each Service RR.¶
The term Service RR does not indicate use of the iBGP route-reflection procedures defined in [RFC4456]. A Service RR is a regular eBGP speaker. It applies the existing BGP procedures defined in [RFC4271], including best-path selection and AS_PATH processing. When advertising a service route to a leaf, its one special behavior is to leave the BGP NEXT_HOP unchanged. The next hop therefore continues to identify the leaf that originated the service route. The receiving leaf resolves that next hop through the infrastructure routes distributed by the spines. If the service route carries an SRv6 Service SID, the Service RR also propagates the associated SRv6 Service TLV, as specified in [RFC9252] for advertisements that retain the BGP next hop.¶
The Service RRs do not forward service traffic. An ingress leaf encapsulates the traffic in IPv6. The outer IPv6 destination identifies an encapsulation destination reachable through the infrastructure routes, such as an egress-leaf address or an SRv6 Service SID. The spines forward the encapsulated packet using infrastructure reachability.¶
The benefit of this design is a reduction in service-route instances at each leaf and the removal of service routes and their update processing from the spines.¶
This document does not define new BGP extensions or data-plane behavior.¶
The following terms are used in this document:¶
BGP route: A service or infrastructure prefix together with the BGP path attributes associated with that prefix.¶
BGP route instance: A BGP route received from a particular peer. A leaf can receive multiple route instances for the same prefix from different peers.¶
Infrastructure prefix: A prefix used to reach a switch loopback, link, tunnel endpoint, SRv6 locator, or other part of the fabric infrastructure.¶
Service prefix: A prefix used to reach a server, workload, or service endpoint.¶
Service RR: An eBGP speaker that receives service routes from leaves and advertises selected service routes to other leaves. It uses regular eBGP procedures and preserves NEXT_HOP when propagating a service route. It does not use the iBGP route-reflection procedures of [RFC4456].¶
Consider a full-bandwidth two-tier folded Clos in which each spine has one link to each leaf. This document uses rounded values to make the calculations easier to follow:¶
S = 200 spines L = 500 leaves P = 200 service endpoint prefixes per leaf Q = 4 Service RRs¶
Assuming one infrastructure prefix per switch, the fabric contains 700 infrastructure prefixes. Assuming one service prefix per endpoint, the fabric contains 100,000 service prefixes, corresponding to 500 leaves with 200 endpoint prefixes per leaf in this example.¶
Following the ASN-allocation style described in [RFC7938], this document assumes that all spines use a common ASN and that every leaf uses a distinct ASN. The common spine ASN prevents a route received from one spine from being advertised by a leaf to another spine, because the receiving spine's ASN would already be present in the AS_PATH.¶
Servers do not need to run BGP in this model. A leaf can learn locally attached service reachability by connected or static routing and originate the corresponding service routes into BGP.¶
Every leaf has a direct eBGP session with every spine. Each leaf originates and advertises a route for its infrastructure prefix to every spine. A spine advertises a leaf's infrastructure route to all other leaves. Each spine originates and advertises a route for its own infrastructure prefix to every leaf.¶
For S = 200 and L = 500, there are 700 infrastructure prefixes: 200
spine prefixes and 500 leaf prefixes.¶
Since a spine receives one leaf infrastructure route from each of the
L leaves, it holds L received leaf-originated routes, in addition
to its locally originated infrastructure route.¶
A leaf receives one directly originated spine route from each of the
S spines. It also receives one route from each spine for every other
leaf's infrastructure prefix. Its received infrastructure-route count
is therefore:¶
S + S * (L - 1) = S * L¶
For S = 200 and L = 500, a leaf receives 100,000 BGP route
instances from its neighbors. These route instances represent 699
remote infrastructure prefixes: 200 spine prefixes and 499 other-leaf
prefixes.¶
When supported and enabled, BGP multipath allows the leaf to install multiple ECMP paths for a remote leaf prefix in the forwarding table. The leaf could install the 499 other-leaf infrastructure prefixes with 200 ECMP paths per prefix.¶
In the existing design, not using Service RRs, each leaf advertises one route
for each of its P locally attached service endpoint prefixes to
each spine. A spine receives one route for every service prefix, for a
total of:¶
L * P¶
With L = 500 and P = 200, each spine receives 100,000 service
routes. It advertises each service route to the other leaves.¶
A leaf receives one route from every spine for every service prefix originated by another leaf. Its received remote service-route count is:¶
S * (L - 1) * P¶
For the example topology:¶
200 * (500 - 1) * 200 = 19,960,000 route instances¶
The leaf receives 19,960,000 remote BGP route instances representing 99,800 remote service prefixes. Together with its 200 locally attached service prefixes, the leaf represents 100,000 unique service prefixes.¶
When supported and enabled, BGP multipath allows the leaf to install multiple ECMP paths for a remote service prefix in the forwarding table. The leaf could install the 99,800 remote service prefixes with 200 ECMP paths per prefix.¶
The Service RR design separates infrastructure-route distribution from service-route distribution:¶
The existing leaf-spine eBGP sessions distribute infrastructure routes.¶
Leaves do not advertise service routes to the spines, and the spines do not hold or advertise service routes.¶
Each leaf establishes an additional multihop eBGP session with each Service RR.¶
Leaves advertise service routes to the Service RRs.¶
Each Service RR distributes eligible service routes to the leaves.¶
Import and export policies enforce this separation. Infrastructure routes remain available to resolve BGP next hops and IPv6 encapsulation destinations and to forward encapsulated traffic across the fabric.¶
The Service RR is a control-plane node. It is not in the forwarding path between leaves.¶
A Service RR is a regular eBGP speaker. It performs the existing BGP procedures in [RFC4271].¶
Each Service RR uses an ASN different from every leaf ASN, so every Service RR-to-leaf session is an eBGP session. Service RRs can share an ASN or use different ASNs. This choice can affect multipath eligibility but does not change the route-distribution procedures described here.¶
The behavior specific to this architecture is that the Service RR advertises a service route with NEXT_HOP unchanged. The NEXT_HOP received from the originating leaf is therefore retained when the route is advertised to another leaf. The receiving leaf resolves that next hop through the infrastructure routes learned from the spines.¶
If a service route carries an SRv6 Service SID, the Service RR also propagates the associated SRv6 Service TLV, as specified in [RFC9252] for advertisements that retain the BGP next hop. When present, the receiving leaf resolves the Service SID through the infrastructure route that covers its locator.¶
When a route for an IPv4 service prefix is advertised with the originating leaf's IPv6 address as NEXT_HOP, the peers use the Extended Next Hop Encoding capability and procedures specified in [RFC8950]. The Service RR preserves that IPv6 next hop when advertising the service route to another leaf. This next-hop encoding applies independently of the encapsulation selected for the service traffic.¶
Service RRs operate independently and do not need to exchange service routes with one another because each leaf advertises its service routes to each Service RR.¶
Each leaf establishes a multihop eBGP session with each of the Q
Service RRs. Every Service RR receives the service routes directly
from the originating leaves, independently selects its best routes,
and advertises eligible routes to the other leaves.¶
The Service RRs should reside in separate failure domains. No Service RR is a data-plane next hop, so the loss of one Service RR does not directly affect service traffic forwarding, provided that at least one other Service RR remains available and advertises equivalent usable routes.¶
Adding Service RRs increases the number of BGP sessions at a leaf. In
the example, a leaf retains its 200 sessions to the spines and adds
Q Service RR sessions. With four Service RRs, a leaf has 204 BGP
sessions in total.¶
Each Service RR receives one route for each of the L * P service
prefixes. In the example, each Service RR receives 100,000 service
routes.¶
Each Service RR advertises one selected route for a remote service
prefix to a leaf. A leaf can therefore receive Q route instances for
each remote service prefix. Its received remote service-route count is:¶
Q * (L - 1) * P¶
With four Service RRs (Q = 4):¶
4 * (500 - 1) * 200 = 399,200 route instances¶
Compared with the design without Service RRs, the number of remote service-route instances received by each leaf is significantly reduced. In this example, it is reduced from 19,960,000 to 399,200, a factor of 50. The number of unique service prefixes and associated FIB entries for service prefixes at a leaf remains approximately 100,000.¶
The Service RR design reduces replicated BGP route state and update processing at leaves and removes service-route state and update processing from the spines.¶
The spine routing tables do not contain routes for service prefixes. An ingress leaf therefore encapsulates a service packet in an outer IPv6 packet before sending it across the fabric. The spines forward the outer IPv6 packet using infrastructure routes. The encapsulation determines how the outer destination is selected and resolved.¶
This document describes IP-in-IPv6 and SRv6 encapsulation as examples.¶
For IP-in-IPv6 encapsulation [RFC2473], the BGP NEXT_HOP of the service route identifies the egress leaf rather than the Service RR. The ingress leaf recursively resolves this next hop through an infrastructure route to the egress leaf. That infrastructure route can provide equal-cost paths through multiple spines.¶
The ingress leaf places the original IPv4 or IPv6 packet in an outer IPv6 packet whose destination is an IPv6 address assigned to the egress leaf, such as its loopback address. The egress leaf removes the outer IPv6 header and forwards the inner packet to the service endpoint.¶
For SRv6 encapsulation, an egress leaf allocates an SRv6 Service SID for the applicable forwarding context and associates that SID with the advertised service routes. For a global IPv4 and IPv6 service, the Service SID can be an End.DT46 SID associated with the main IPv4 and IPv6 forwarding tables, as specified in [RFC8986]. The egress leaf advertises the service route and its Service SID using the SRv6 L3 Service TLV of the BGP Prefix-SID attribute, as specified in [RFC9252].¶
The Service SID is allocated from an SRv6 locator for which the egress leaf advertises an infrastructure route. The ingress leaf uses the Service SID as the destination of the outer IPv6 header, and the spines forward the packet using the route for the locator. The BGP NEXT_HOP continues to identify the egress leaf and is used to track reachability to that leaf.¶
At the egress leaf, the Service SID behavior processes the packet. If the Service SID is an End.DT46 SID, the egress leaf decapsulates the packet and performs the IPv4 or IPv6 lookup.¶
A service route remains an individual route regardless of the encapsulation. It is usable only while its BGP next hop and the forwarding information required by the selected encapsulation remain resolvable.¶
A Service RR does not appear in the forwarding chain. Loss of a single Service RR therefore does not directly break the data plane. The affected eBGP session fails, routes learned only over that session are withdrawn, and regular BGP route selection chooses among the routes remaining from other Service RRs. Forwarding is expected to continue when at least one other Service RR remains established and advertises an equivalent usable route.¶
This document has no IANA actions.¶
The authors thank Ahmed Abdelsalam for his input and review.¶