Lightspeed Notification Protocol

Introduction Artificial Intelligence (AI) and Machine Learning (ML) workloads impose stringent demands on data center fabrics, and in some deployments on the Data Center Interconnect (DCI) and WAN that join those fabrics. As outlined in , these environments require network convergence times in the sub-100 microsecond range to avoid performance degradation caused by packet loss and jitter. Traditional routing protocols and convergence mechanisms rely on control-plane intervention, where link state changes are processed by the CPU. This introduces activation latencies in the tens of milliseconds, which is orders of magnitude too slow for these requirements. Furthermore, existing mechanisms typically operate on a binary "up/down" model and do not signal capacity degradation or congestion. This document defines the Lightspeed Notification Protocol (LSN), a hardware-accelerated notification mechanism that operates on chip to handle both network failures and congestion events. LSN is complementary to traditional routing protocols (for example, BGP). The interaction between the routing protocol and LSN is functionally equivalent to a boolean AND operation: a path is eligible for forwarding only if permitted by the routing protocol AND confirmed healthy by the hardware notification. This enforces an asymmetric reaction. "Bad news" (failure or congestion) acts as an immediate hardware veto, disabling the path in sub-microseconds. "Good news" (recovery) is gated by the routing protocol, ensuring the control plane has fully converged before traffic resumes. This approach combines sub-microsecond protection with routing coherence and stability. LSN defines two encoding modes. Dense encoding () is used in DC fabrics, where ECMP is always present through many spines. Sparse encoding () is used in the DCI and WAN. The encodings differ due to the different network characteristics of those environments.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Terminology

LSN: Lightspeed Notification Protocol.
Dense encoding: The LSN encoding used in DC fabrics, where the destination set is the other leaves and forwarding is ECMP across many spines.
Sparse encoding: The LSN encoding used in the DCI and WAN, where the destination set is sparse and large and forwarding may be ECMP, weighted ECMP, or primary/backup.
DC: Data Center fabric, typically a Clos spine-leaf topology.
DCI: Data Center Interconnect, the network that joins DC fabrics.
WAN: The wide-area network that carries traffic between regions or sites of an AI/ML fabric.
uN: The uSID function defined in . In the context of LSN, uN identifies the node advertising the LSN event.
uA: The uSID function defined in . In the context of LSN, uA identifies the link or bundle member associated with the LSN event.
Efficient Remote Protection: The IP Fast Reroute mechanism specified in , also known as remote TI-LFA. It activates a pre-computed backup at a node upstream of the failure to avoid hairpinning.

Problem Statement Current IP Fast Reroute (IPFRR) and resiliency mechanisms face fundamental limitations when a remote hop fails or congests behind a next-hop that remains up. Latency of CPU-Based Processing Existing failure detection and propagation mechanisms generally trigger interrupts that must be processed by a line-card or system CPU. As noted in , this host-CPU path introduces an activation delay typically in the range of 10-50 milliseconds. For AI training workloads requiring barrier synchronization, this latency results in unacceptable flow disruption. To achieve the required sub-100 microsecond convergence, the propagation and processing of network state changes MUST occur on chip and MUST NOT be punted to the host CPU. Remote Failure Behind a Live Next-Hop A common failure mode is that the ingress node's immediate next-hop remains reachable while a downstream link has failed. In a DC Clos fabric, traditional BFD/TWAMP liveness of the BGP session to the spine remains up, so local detection and PIC-core at the ingress leaf do not fire. In a DCI or WAN, traditional BFD/TWAMP liveness of the BGP session to the next-hop likewise remains up, so local FRR does not activate. In both cases the control plane will eventually withdraw the affected prefixes, but traffic blackholes for the entire control-plane window. The final FIB produced by BGP is already correct; the issue is that the dataplane sits wrong until that reconvergence completes. Inability to Handle Congestion and Partial Failures Standard routing protocols and ECMP operate on a binary failure model: a link is either available or unavailable. AI fabrics, including the DCI and the WAN, frequently experience brownout scenarios, such as partial capacity reduction (for example, loss of a single member in a bundle) or acute congestion. Current mechanisms lack the granularity to signal these states, causing traffic to be blackholed or hashed onto congested paths. When the BGP next-hop represents a LAG, the safe reaction without per-destination visibility is to disable the whole bundle, taking remaining healthy members out of service. Limited Visibility of Remote Network State In multi-stage Clos topologies and in DCI and WAN topologies, a failure or congestion event often occurs multiple hops away from the ingress point. Local protection at the point of failure is often insufficient, or it leads to suboptimal hairpin routing that increases latency and wastes bandwidth. Efficient Remote Protection addresses the hairpin by activating a backup at an upstream node, but it depends on a timely forwarding-plane notification of the remote event. Current routing protocols do not provide that notification in the required time budget.

Common Operation LSN is not a routing protocol and is not a BGP attribute. It is an on-chip notification, generated and processed without the host CPU, that overlays a health signal on the paths already installed by the routing protocol. The following properties apply to both the dense and the sparse encodings:

A node monitors the status of its local interfaces in hardware. Upon a local link failure, capacity change, or congestion-level change, it originates an LSN notification. Local origination is based on interface status and local congestion state, not on traditional BFD/TWAMP liveness.
A single notification packet is pre-crafted and ready to send. Crafting a distinct packet per neighbor at the time of the event introduces unacceptable delay. The hardware rewrites the fields that identify the event and replicates the packet on the relevant interfaces.
A receiving node MUST process the notification on chip. The packet MUST NOT be punted to the host CPU.
A path is usable if and only if the routing protocol has installed it AND LSN reports it healthy. Bad news is a hardware veto. Good news waits for the routing protocol.
LSN does not recompute routes. In dense encoding, used in DC fabrics, it masks members of an existing ECMP set. In sparse encoding, used in the DCI and WAN, it eliminates a member from an ECMP set, changes the weighted-ECMP (wECMP) weights, or activates a pre-computed backup (for example an Efficient Remote Protection / remote TI-LFA path).

Two encodings are defined, due to the different network characteristics of DC fabrics and of the DCI and WAN. specifies the dense encoding and illustrates it in a DC Clos. specifies the sparse encoding and illustrates it in a DCI/WAN topology.

Dense Encoding This section specifies the dense encoding. Dense encoding is used in DC fabrics, where ECMP is always present through many spines and the destination set is the other leaves of the fabric. The notification carries a bitmap of destination reachability or congestion, and the ingress leaf overlays that bitmap on the spine next-hops already installed by BGP.

Dense Notification Packet Format The dense notification is an Ethernet MAC Control frame. The packet header is the following:

Dense LSN Ethernet header Notes:

MAC Destination Address: 01-80-C2-00-00-01 (IEEE MAC-specific control protocols)
Ethertype = TBD1
Opcode = TBD2
MAC Source Address: the neighbor that originated the notification. The receiver uses the source address to identify which spine next-hop (column) to update.

The LSN Payload is defined as follows:

Dense LSN payload Field Descriptions:

Header Type (4 bits): Value '12' - Identifies this as a topology reachability packet.
R (1 bit): Reserved. It MUST be set to zero upon transmission and ignored upon receipt.
Msg-type (2 bits): Indicates the information conveyed by the bitmap. It may represent reachability or congestion status. The congestion status is defined as 3 different congestion levels, which are implementation specific. The following Msg-Types are defined:
- 0x0: Reachable
- 0x1: Reachable and not congested (Congestion Level 1)
- 0x2: Reachable and not congested (Congestion Level 2)
- 0x3: Reachable and not congested (Congestion Level 3)
Reserved (3 bits): Reserved. It MUST be set to zero upon transmission and ignored upon receipt.
Reachable-Devices-Range (6 bits): Determines the range of the reachable devices bitmap. The range per packet is within [Reachable-Devices-Range*256, Reachable-Devices-Range*256 + 255].
- '0': Devices 0-255
- '1': Devices 256-511
- '2': Devices 512-767
- ... (Covers up to 16K devices).
Reachable-Devices-Bitmap (256 bits): One bit per destination leaf. Bit i indicates whether destination leaf i is healthy through the device sending this message.

Dense Node Identification To facilitate bitmap-based signaling, every leaf node within the fabric is assigned a unique Global Node ID. This identifier maps the node to a specific bit position within the notification bitmap. The allocation, synchronization, and lifecycle management of these Global Node IDs are performed by an external central controller or management plane. The specific mechanisms for ID allocation are outside the scope of this document.

Dense Origination Each spine is responsible for generating notification messages that convey the direct reachability or congestion state of the leaves attached to it. Message Scope and Fragmentation A single notification message supports a payload covering up to 256 destination devices.

The message header MUST specify the Reachable-Devices-Range (or Base ID), indicating the starting Global Node ID for the bitmap payload.
If the fabric scale exceeds 256 leaves, the originating node MUST generate multiple distinct notification messages. Each message corresponds to a different, non-overlapping Reachable-Devices-Range to cover the full topology.

MSG-Type and State Encoding The interpretation of the bitmap payload is determined by the MSG-Type field in the header.

Reachability Information: When the MSG-Type indicates reachability, the bitmap encodes the binary status of the downlinks:
- Bit value 1: Indicates the destination leaf is directly reachable.
- Bit value 0: Indicates the destination leaf is not reachable.
Congestion Information: When the MSG-Type indicates congestion, the bitmap conveys the congestion status of the associated downlinks. The mapping of queue congestion levels to the various congestion values is implementation-specific, provided it supports the hardware logic required for path pruning or de-prioritization.

Notifications are event-driven. One is emitted when an interface changes state or a path's congestion level changes significantly; the originating node does not restate the whole fabric on every tick. A low-rate periodic refresh sits underneath those events, so that a switch that missed a notification resynchronizes without waiting for the next change.

Dense Hardware Processing Upon receipt of a dense notification message, the receiving leaf MUST process the packet on chip. The packet MUST NOT be punted to the host CPU. State Aggregation The receiving leaf maintains a local health table with one row per destination leaf and one column per spine next-hop. When a message is received:

The hardware identifies the column from the MAC Source Address (the advertising spine) and the row-range from the Reachable-Devices-Range.
The received bitmap is overlaid onto that column, updating the reachability or congestion status for the corresponding destination leaves.

Forwarding Decision Logic To derive the final forwarding decision, the hardware performs a bitwise operation between the standard routing table and the aggregated notification state.

For reachability, this is functionally a Bitwise AND operation: a path is valid if and only if the routing protocol has installed it (Logical 1) AND the notification protocol reports it as reachable (Logical 1).
If the result of this operation is 0 for a specific next-hop, the hardware immediately removes that next-hop from the Equal-Cost Multi-Path (ECMP) set for that destination. Next-hops used for other destinations are unchanged.

When a bundle (LAG) is used toward a spine, the health table MAY be maintained per bundle member. A single member failure then reduces capacity toward the affected destination (for example from 100% to 75% on a 4-member bundle) rather than withdrawing the entire BGP next-hop. The BGP adjacency remains up; LSN only changes membership or weight for the destinations that use the failed member.

Illustration: Dense Encoding in a DC Fabric This section illustrates dense encoding in a DC Clos fabric. The example uses three spines and three leaves so that the topology is readable; the same procedure applies to a fabric of 256 spines and 256 leaves.

DC Topology Spines are identified by letters. Leaves are identified by numbers. Every leaf is connected to every spine.

Spines: A, B, C.
Leaves: 1, 2, 3. Each leaf's Global Node ID equals its number.
Traffic: leaf 1 sends traffic to leaf 3 (and also to leaf 2).
Routing state: BGP advertises that leaf 3 is reachable via all spines. Leaf 1 maintains ECMP {A, B, C} for destination 3, and the same ECMP for destination 2.

DC Clos fabric used in the illustration

DC Failure Scenario A physical link failure occurs between spine B and destination leaf 3.

Detection and Generation: Spine B detects that the port connected to leaf 3 is down. The hardware at spine B immediately generates a notification packet and transmits it to all connected leaves, including leaf 1.
- MAC Destination Address: 01-80-C2-00-00-01 (MAC Control)
- MAC Source Address: spine B
- Reachable-Devices-Range (Base ID): 0 (covering leaves 0-255, here 1, 2, and 3).
- MSG-Type: Reachability.
- Bitmap: three entries, one per attached leaf. 1 and 2 remain active (1). The third entry, Leaf 3, is set to inactive (0).

Next-hop vector for spine B after failure of B-3

Hardware Processing at Ingress: Leaf 1 receives the notification from spine B. The forwarding engine overlays column B of its health table and performs a logical AND with the BGP ECMP set.
- Destination 3 via spine B:
  - Routing table state for leaf 3: 1 (the control plane has not yet converged; leaf 3 is still considered reachable via B).
  - Notification state from spine B for leaf 3: 0.
  - Result: 1 AND 0 = 0 (path invalid).
- Destination 3 via spines A and C:
  - Routing table state for leaf 3: 1.
  - Notification state from A and from C for leaf 3: 1.
  - Result: 1 AND 1 = 1 (path valid).
- Destination 2 via spine B:
  - The bit for leaf 2 in the same notification is 1. Traffic from leaf 1 to leaf 2 continues to use B. Spine B is not drained.
Forwarding Result: Because the result for spine B is 0 only for destination 3, the forwarding hardware excludes B from the ECMP group for leaf 3. Traffic to leaf 3 is rebalanced across {A, C}. Balancing of traffic to leaf 2 is unchanged: {A, B, C}.

This mechanism provides sub-microsecond protection. When the link B-3 is restored, spine B sends an updated bitmap with the leaf-3 bit set to 1. Traffic does not resume via B until BGP also re-installs the route, satisfying the 1 AND 1 condition.

Sparse Encoding This section specifies the sparse encoding. Sparse encoding is used in the DCI and WAN, where the destination set is sparse and large. Forwarding may be an ECMP set, a weighted-ECMP (wECMP) set, or a primary/backup pair; it is not limited to the wide spine ECMP of a Clos fabric. The notification identifies the event type, the node that detected the event, and the affected adjacency. A node that has registered interest in that adjacency either eliminates a member from an ECMP set, changes the wECMP weights, or activates a pre-programmed backup, most commonly an Efficient Remote Protection (remote TI-LFA) path . LSN does not compute that ECMP change or that backup. An agent with visibility of the topology programs the ECMP membership, the wECMP weights, and the backup pointers before any failure. LSN only sets the protection state that the agent has already installed.

Sparse Notification Packet Format The sparse notification is an Ethernet frame carrying an IPv6 header and no payload.

Sparse LSN packet The IPv6 header is used as follows:

Source Address: an IPv6 address of the node originating the notification.
Destination Address: the LSN Notification SID of the domain. 2001:db8::/32 is the documentation prefix standing in for the SRv6 uSID block of the domain . This SID is associated with the LSN Endpoint Behavior (TBD3). The LSN Endpoint Behavior takes three 16-bit arguments, in order:
- Event-Type: the kind of event being advertised.
- uN: the uSID function defined in ; in LSN it identifies the advertising node.
- uA: the uSID function defined in ; in LSN it identifies the affected link or bundle member.
In this document's illustrations, FF00 is a SID associated with the LSN Endpoint Behavior. The resulting SID is structured as 2001:db8:FF00:Event-Type:uN:uA:: (see ).
Payload Length: 0.
Next Header: 59 (IPv6 No Next Header).
Hop Limit: the blast radius of the notification.

LSN Notification SID The Event-Type argument indicates the information conveyed by the notification. Value 0x0000 is reserved. The following values are defined:

0x0001: Down / not reachable
0x0002: Up / recovered
0x0003: Congestion Level 1
0x0004: Congestion Level 2
0x0005: Congestion Level 3

Congestion levels are implementation specific. When Event-Type indicates congestion, a registered node MAY change wECMP weights or de-prefer the adjacency; it is not required to treat the adjacency as failed. This document uses the following identifier convention in its illustrations:

Event-Type uses the values defined above. The failure illustration uses Event-Type = 0001 (Down).
uN is 0X00, where X is the node number of the advertising node. For example, node 2 uses uN = 0200.
uA is F0YZ, where Y is the source node of the adjacency and Z is the destination node of the adjacency. For example, the link from node 2 to node 3 uses uA = F023.

The allocation of uN and uA values in a deployment is performed by the operator or by a controller. The specific allocation protocol is outside the scope of this document.

Sparse Origination Every node monitors its local links. Upon detecting a local link failure, a link-up event, a congestion-level change, or a change of link properties (for example a bundle capacity change that must be reflected in weighted ECMP), the node originates a sparse LSN notification. The originating hardware keeps one pre-crafted IPv6 notification ready to send. Distinct packets per neighbor MUST NOT be required. When the local interface-status or congestion logic detects the event, it writes the LSN Notification SID (a SID associated with the LSN Endpoint Behavior, together with the three arguments Event-Type, uN, and uA) and sends that same packet out on all interfaces. Local origination is based on interface status registers (on the order of 10 microseconds), not on traditional BFD/TWAMP liveness. Traditional BFD/TWAMP liveness continues to serve the BGP session; it is not the trigger for LSN.

Sparse Processing and Registration Upon receipt, the packet is processed in the dataplane. A hardware classifier matches the LSN Notification SID (domain uSID block and the SID associated with the LSN Endpoint Behavior). The match triggers two actions:

If the Hop Limit is greater than 1, decrement the Hop Limit and forward the notification on the interfaces indicated by the local LSN forwarding policy, so that registered upstream nodes can receive it.
Process a copy of the packet on chip. The LSN Endpoint Behavior (TBD3) is invoked with its three arguments (Event-Type, uN, uA) to activate any locally registered protection for that tuple.

an agent, before any failure, programs an exact-match table:

Key: the three arguments of the LSN Endpoint Behavior (Event-Type, uN, uA).
Value: a linked list of local protection pointers. Each pointer identifies a FIB protection bit to set: elimination of an ECMP member, a wECMP weight update, or activation of an Efficient Remote Protection (remote TI-LFA) backup.

A single DCI or WAN adjacency often covers many destinations. The linked list allows one notification to enable every locally relevant protection entry without a host-CPU walk of the RIB. Failure and congestion on the same adjacency use different Event-Type values and therefore different table entries, so they MAY install different protection lists. The lookup on (Event-Type, uN, uA) returns the first protection pointer and the next-member link. The protection pointer is set on chip and the list is walked until it terminates. The packet MUST NOT be punted to the host CPU. LSN does not compute the ECMP change or the backup path. The agent has already installed the ECMP members, the wECMP weights, and any Efficient Remote Protection (remote TI-LFA) backup . LSN sets the protection state that applies them.

Illustration: Sparse Encoding in the DCI and WAN This section illustrates sparse encoding in a DCI/WAN topology. The example uses a primary/backup pair and Efficient Remote Protection. The same notification can instead eliminate a member of an ECMP set or change wECMP weights. Routers are identified by numbers. Numbers are not reused.

DCI/WAN Topology Consider the topology in .

Node 1 is the ingress / Point of Remote Repair.
The primary path from 1 to destination 7 is 1-2-3-7.
The backup path from 1 to destination 7 is 1-4-5-6-7.
The protected resource is the adjacency 2-3.

Identifier assignment for the illustration:

Node 1: uN = 0100
Node 2: uN = 0200
Node 3: uN = 0300
Node 4: uN = 0400
Node 5: uN = 0500
Node 6: uN = 0600
Node 7: uN = 0700
Adjacency 2-3: uA = F023
Failure of adjacency 2-3: Event-Type = 0001

DCI/WAN topology used in the illustration

DCI/WAN Failure Scenario Traffic from node 1 to destination 7 uses next-hop 2. Local FRR at node 1 does not fire, because the protected resource is the remote adjacency 2-3. If only local TI-LFA were used at node 2, traffic already at node 1 would continue toward node 2 and would then be repaired in a way that hairpins back through the upstream path. Efficient Remote Protection programs the repair at node 1: if adjacency 2-3 fails, traffic from 1 to 7 is steered on 1-4-5-6-7.

Pre-failure programming: The agent installs at node 1:
- An Efficient Remote Protection backup for destination 7, protecting adjacency 2-3, via 4-5-6.
- An exact-match entry for (Event-Type=0001, uN=0200, uA=F023) whose protection pointer enables that backup.
- On node 2, a Hop Limit of 1.
Detection and Generation: Node 2 detects that the local port toward node 3 is down. The hardware originates one IPv6 LSN notification:
- Source Address: node 2.
- Destination: LSN Notification SID 2001:db8:FF00:0001:0200:F023::.
- Hop Limit: 1.
- Payload Length: 0; Next Header: 59. The packet is replicated in hardware on the interfaces of node 2, including the interface toward node 1.
Processing at node 1: The classifier matches the LSN Notification SID. Node 1 invokes the LSN Endpoint Behavior (TBD3) with arguments (0001, 0200, F023), walks the protection linked list, and sets the Efficient Remote Protection pointer for destination 7. Traffic from 1 to 7 is switched onto 1-4-5-6-7. The packet MUST NOT be punted to the host CPU.
AND with routing: The backup is used only while BGP still authorizes a path from 1 to 7 and LSN reports adjacency 2-3 as failed. When 2-3 is restored, node 2 may originate a corresponding recovery notification with Event-Type = 0002 (Up). Traffic returns to 1-2-3-7 only after BGP has reconverged, satisfying the AND condition.

The same notification can enable more than one protection pointer. If adjacency 2-3 is also a weighted-ECMP input for other destinations at node 1, those destinations are chained from the same (0001, 0200, F023) entry. A congestion event on the same adjacency uses a different Event-Type (0x0003, 0x0004, or 0x0005) and therefore a different exact-match entry, which MAY update wECMP weights without activating the backup.

Security Considerations The Lightspeed Notification Protocol (LSN) is designed to achieve sub-100 microsecond convergence on chip. To achieve this performance, LSN messages do not include cryptographic authentication or integrity checks. Consequently, the protocol introduces several security considerations that must be mitigated by the deployment architecture. Spoofing and Denial of Service (DoS): Because LSN messages are unauthenticated, an attacker capable of injecting forged LSN frames into the fabric could advertise false congestion or failure notifications. This would cause receiving nodes to prune valid paths or to activate backups, potentially leading to forced congestion, suboptimal routing, or a denial of service. Mitigation via Boundary Filtering: LSN is intended exclusively for internal use within a single administrative domain (a DC fabric and, where deployed, that operator's DCI and WAN). To prevent spoofing, boundary nodes MUST implement strict port-level filtering. LSN notification packets MUST be dropped unconditionally on any port facing a server, host, or external network. LSN processing MUST only be enabled on trusted infrastructure links. In the sparse encoding, the Hop Limit further constrains how far a notification can travel. Fail-Safe Routing Logic: As defined in this document, LSN acts only as a hardware veto (a logical AND operation with the routing protocol). While an attacker can maliciously prune a path or force a pre-authorized backup, they cannot force the network to forward traffic into a blackhole or an invalid topology segment, because reachability still requires the routing protocol (for example, BGP) to authorize the path. The control plane remains the source of truth for topology validation.

IANA Considerations This document requests the following allocations.

Ethertype: A new Ethertype for the Lightspeed Notification Protocol (LSN) dense encoding (TBD1).
Opcode: A new MAC Control Opcode for LSN dense bitmap reachability (TBD2).

IANA is requested to allocate a codepoint in the "SRv6 Endpoint Behaviors" registry as follows: SRv6 Endpoint Behavior allocation

Value	Hex	Endpoint Behavior	Reference	Change Controller
TBD3		LSN Notification	This document	IETF

Upon assignment of these values, the RFC Editor is requested to replace all instances of TBD1, TBD2, and TBD3 with the allocated values.

Running Code The dense encoding is running on Cisco Silicon One G300. Notification generation and notification processing are performed entirely on chip, at linerate. LSN notifications are not processed on the host CPU and are not handed to the control plane. The prototype instantiates the IANA codepoints of this document as follows:

TBD1 (Ethertype): 0x8808 (IEEE 802.3 MAC Control )
TBD2 (MAC Control Opcode): 0x5AA5

Acknowledgements The authors would like to acknowledge Kris Michielsen.