China Mobile

Beijing 100053 China yangjinwl@chinamobile.com

China Mobile

Beijing 100053 China hantingting@chinamobile.com

China Mobile

Beijing 100053 China chengweiqiang@chinamobile.com

China Mobile

Beijing 100053 China tianyuchi@chinamobile.com

Centec

Suzhou 215000 China wangjj@centec.com

Centec

Suzhou 215000 China zhanggy@centec.com

RTG RTGWG MC-LAG inter-chassis link redundancy group forwarding continuity failure recovery When a downlink member of a Multi-Chassis Link Aggregation Group (MC-LAG) fails, a common recovery method flushes the MAC forwarding table and relies on flooding to re-establish the forwarding paths, which can cause a period of congestion and disruption. This document specifies two complementary forwarding-plane failure recovery procedures for MC-LAG. The first procedure preserves the forwarding state by adjusting the LAG group's member set, adding an inter-chassis link (ICL) port in place of the failed downlink, so that MAC table entries remain valid and traffic is redistributed without a table flush and without flooding. The second procedure extends recovery beyond a single Provider Edge (PE) pair: when the immediate peer cannot satisfy a failover request, the request is relayed over a secondary ICL to the next PE along a configured relay path, which allows recovery across three or more PEs in the redundancy domain. The procedures operate primarily in the forwarding plane and do not require Ethernet VPN (EVPN) infrastructure.

Introduction Multi-Chassis Link Aggregation Group (MC-LAG) technology allows two or more Provider Edge (PE) devices to present a single Link Aggregation Group (LAG) interface toward a Customer Edge (CE) device . An inter-chassis link (ICL, also called a peer link) connects the PE devices and carries both Inter-Chassis Communication Protocol (ICCP) signaling and redirected data traffic. When a downlink member fails, a common recovery method flushes the MAC address table entries associated with the affected LAG group and relies on flooding of unknown unicast to relearn the forwarding paths. In a large deployment the flooding phase can last for hundreds of milliseconds and cause congestion and application-level disruption. EVPN-based solutions reduce this through control-plane mass-withdraw signaling, but require EVPN infrastructure. This document specifies two procedures that operate primarily in the forwarding plane and do not require EVPN infrastructure. The first procedure, forwarding state preservation (), responds to the removal of a downlink member from a LAG group by adding an ICL port to the same group. Because MAC table entries reference the LAG group identifier rather than an individual port, the entries remain valid and traffic is redistributed by the LAG hash function without a table flush and without flooding. The second procedure, ICL relay (), applies in topologies with three or more PEs. A PE that receives a failover request but cannot satisfy it locally relays the request over a secondary ICL to the next PE. The relay continues hop by hop until a PE with an operational downlink is reached. The two procedures are independent and can be deployed separately or together. When combined, forwarding state preservation handles local failover within a PE pair and ICL relay extends recovery to additional PEs in the redundancy domain. The procedures carry MUST-level requirements on PE forwarding and signaling behavior and request an IANA code point for the relay messages, and so this document is on the Standards Track.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Terminology This document uses the terminology of and . The following terms are defined.

Downlink Member:

A port that belongs to an MC-LAG group and connects to a downstream CE device.

ICL:

Inter-Chassis Link (peer link). A link or LAG that connects two PEs in an MC-LAG redundancy group (RG).

Primary ICL:

The ICL between the two PEs that form the normal MC-LAG peer pair for a given RG.

Secondary ICL:

An ICL between two PEs that are not direct MC-LAG peers for a given RG. A secondary ICL provides a relay path toward additional PEs.

LAG Group ID:

The forwarding-plane identifier of a LAG group. MAC table entries reference this identifier, and the forwarding engine selects an egress member by a hash function.

Member Set:

The current set of physical ports that belong to a given LAG group.

Relay Path:

An ordered set of PEs and ICLs along which a failover request may be forwarded, configured administratively on each participating PE.

Downstream Isolation:

A condition in which a PE has lost all operational downlink members for a given MC-LAG group.

Reference Topology The topology in is the reference for both procedures. PE1 and PE2 are connected by a primary ICL, and PE2 and PE3 by a secondary ICL. Each PE has downlink members toward the CE.

Reference Topology -> G1 ]]>

Forwarding State Preservation This procedure preserves the forwarding state during a downlink member failure by adjusting the LAG group's member set.

Preconditions The ICL port MUST be pre-provisioned with the VLAN membership, quality-of-service (QoS) policies, and access control lists (ACLs) of every MC-LAG group for which it may serve as a member. This ensures that adding the ICL to a member set requires only a forwarding-plane table update. A configurable threshold determines when the ICL is added to the member set. The default threshold is the loss of all local downlink members. An implementation MAY allow the operator to specify a minimum local member count below which the ICL is added.

Failover When a PE detects a link-down event on a downlink member, it removes the failed port from the affected LAG group's member set. If the remaining local member count falls below the configured threshold, the PE MUST add one or more ICL ports to the member set. The LAG Group ID does not change. MAC table entries continue to resolve to the same group, and the hash function redistributes traffic across the updated member set. A flow hashed to the ICL reaches the peer PE, which performs its own lookup and forwards the flow over its operational downlinks.

Forwarding State Preservation {m2, ICL} MAC table: -> G1 (UNCHANGED) Hash over {m2, ICL}: some flows via m2, some via ICL Flows via ICL -> PE2 -> forwarded via m3/m4 ]]>

After modifying the member set, the PE SHOULD notify its MC-LAG peer over ICCP that the ICL now carries redirected traffic for the affected group. The peer uses this notification to adjust split-horizon behavior: traffic that arrives from the ICL for this group MUST be eligible for forwarding out local downlinks and MUST NOT be sent back to the ICL.

Reversion When a failed downlink recovers, the PE adds the recovered port back to the member set and removes the ICL. The LAG Group ID does not change and no MAC table flush occurs. The PE SHOULD notify its peer that the ICL is no longer carrying redirected traffic. A configurable hold timer, with a default of 1 second, SHOULD delay reversion to prevent repeated membership changes when the downlink flaps.

Traffic Handling For known unicast, the MAC table entry resolves to the LAG Group ID and the hash function distributes frames across the current member set. No relearning is needed. For broadcast, unknown unicast, and multicast (BUM) traffic, the ICL is typically already part of the flooding domain in an MC-LAG deployment. Adding the ICL to the member set does not change BUM behavior, but an implementation MUST enforce split-horizon rules to prevent duplicate frames.

ICL Membership State The ICL port maintains a per-MC-LAG-group state.

Standby:

Default. The ICL is not part of the group's member set.

Joining:

Transient. The ICL is being added to the member set.

Active-Member:

The ICL is part of the member set and carries redirected traffic.

Leaving:

Transient. A downlink has recovered and the ICL is being removed from the member set.

ICL Membership State Machine

ICL Relay for Multi-PE Recovery This section specifies procedures for relaying failover requests across multiple PEs when the immediate peer cannot satisfy the request.

Applicability The ICL relay procedures apply to MC-LAG domains comprising three or more PEs where correlated downlink failures can exhaust the failover capacity of the immediate peer pair. Each participating PE MUST have a pre-configured relay path that identifies its secondary ICL and the next-hop PE in the relay sequence.

Failover Request Origination A PE that detects downstream isolation for a given MC-LAG group MUST first attempt local recovery (including the forwarding state preservation procedure of if configured). If the PE remains unable to reach the CE, it originates a Failover-Request message and sends it to its primary MC-LAG peer over the primary ICL. The message carries the originating PE's identifier (Originator-PE-ID), a sequence number, the affected MC-LAG Group ID, and a hop count initialized to one.

Failover Request Processing A PE receiving a Failover-Request evaluates whether it can satisfy the request: If the PE has an available downlink member, it brings the member into service (or confirms it is already operational) and replies with a Failover-Response indicating success. Redirected traffic then flows along the chain of ICLs from the originating PE to this PE. If the PE cannot satisfy the request and a secondary ICL toward a next-hop PE exists, the PE increments the hop count and forwards the Failover-Request over the secondary ICL. If no further relay path exists, or if the hop count has reached the configured maximum (see ), the PE replies with a Failover-Response indicating No-Resource, or the more specific Hop-Limit-Exceeded status when the hop count is the cause.

Relay Path Liveness After a successful relay, each PE along the path MUST exchange periodic Liveness messages with its neighbors on the path. The default interval is 1 second; the default failure threshold is 3 missed messages. If liveness is lost, the PE MUST attempt an alternate path or notify the originator of path failure.

Reversion When the originating PE's downlinks recover, it sends a Revert-Request along the relay path. Each PE restores its ICL segment to standby forwarding state and confirms with a Revert-Response. Reversion SHOULD be delayed by a configurable hold timer (default: 3 seconds).

Message Formats The messages defined in this section support the ICL relay procedures of . They MUST be carried as a new ICCP Application TLV type () over the ICCP connection between adjacent PEs on the relay path, using the registry allocation requested in . Defining the messages over a transport other than ICCP is outside the scope of this document; a future specification that does so MUST preserve the field semantics, the loop prevention rules of , and the security requirements of .

Common Header

Common Message Header

Version (8 bits):

Protocol version. This document defines version 1.

Msg Type (8 bits):

Failover-Request (1), Failover-Response (2), Liveness (3), Revert-Request (4), Revert-Response (5).

Total Length (16 bits):

Length of the entire message in octets.

Originator-PE-ID (32 bits):

Domain-unique identifier of the PE that originated the failover sequence. Used for loop detection.

Sequence Number (16 bits):

Monotonically increasing per originator, for ordering and duplicate suppression.

Hop Count (8 bits):

Incremented at each relay hop. Messages exceeding the configured maximum are discarded.

Reserved (8 bits):

MUST be zero on transmission; ignored on receipt.

Authentication Data (variable):

Optional keyed digest for message integrity, aligned with Section 10.

Message Types and Fields All messages begin with the common header of . The message-specific content is as follows. Failover-Request (Type 1) carries one MC-LAG Group ID TLV () identifying the group that requires recovery.

MC-LAG Group ID TLV

The TLV Type is 1 (MC-LAG Group ID). The Length is the length in octets of the TLV value (4). The MC-LAG Group ID is a 16-bit identifier of the affected group, unique within the redundancy domain. The Reserved field MUST be set to zero on transmission and ignored on receipt. Failover-Response (Type 2) carries an 8-bit Status field followed by the MC-LAG Group ID TLV copied from the corresponding request. The Status values are defined in . Failover-Response Status Values

Value	Name	Meaning
0	Success	A downlink member was brought into service for the group.
1	No-Resource	No PE on the relay path could satisfy the request.
2	Loop-Detected	The request was discarded for loop prevention ().
3	Hop-Limit-Exceeded	The request reached the configured maximum hop count.

Liveness (Type 3) contains only the common header. Revert-Request (Type 4) and Revert-Response (Type 5) each carry the MC-LAG Group ID TLV and manage relay path teardown and restoration. A PE MUST suppress duplicate requests: it maintains, per relay path, the highest Sequence Number seen for each Originator-PE-ID, and MUST silently discard a Failover-Request or Revert-Request whose (Originator-PE-ID, Sequence Number) pair has already been processed. Sequence Number comparison MUST use the serial number arithmetic of to handle wrap-around.

Loop Prevention Two mechanisms bound message propagation in topologies that contain cycles: The Originator-PE-ID field uniquely identifies the PE that began the failover sequence. A PE receiving a Failover-Request carrying its own identifier MUST discard the message and return a Failover-Response with status Loop-Detected. The Hop Count field is incremented at each hop. A PE MUST discard any message whose Hop Count exceeds a locally configured maximum (default: 8) and return a Failover-Response with status Hop-Limit-Exceeded.

Combined Operation When both procedures are deployed together, the following sequence occurs upon a downlink failure on PE1:

Combined Operation Sequence ICL -> PE2 -> secondary ICL -> PE3 -> m5 -> CE. Recovery: m1 link-up PE1 adds m1 to G1, removes ICL. Sends Revert-Request to tear down relay path. ]]>

Per-Group PE States for ICL Relay Each PE maintains the following per-MC-LAG-group state for the relay function:

Operational:

The PE has at least one working downlink member and forwards traffic locally.

Requesting:

The PE has lost all downlink members and has originated a Failover-Request.

Relaying:

The PE has received a Failover-Request that it cannot satisfy locally and has forwarded it along the relay path.

Serving:

The PE has satisfied a relayed request by providing one of its downlink members.

Reverting:

The original failure has been resolved and the relay path is being torn down.

Per-Group PE State Machine

Relationship to Other Protocols

ICCP The procedures in this document do not modify the ICCP state machine defined in . ICCP continues to manage the primary peer relationship. The relay messages are carried in the new ICCP Application TLV type allocated in ; the version negotiation procedures of Section 4.4.1 apply.

EVPN Multi-Homing In deployments using EVPN multi-homing rather than ICCP, the procedures in this document can serve as a forwarding-plane complement. EVPN mass-withdraw provides control-plane convergence; the procedures here provide pre-provisioned data-plane recovery that does not depend on control-plane reconvergence timing.

Operational Considerations

ICL Capacity Planning During failure conditions the ICL carries redirected traffic in addition to its normal load. Operators MUST provision ICL capacity for the worst-case scenario. The recommended capacity is at least equal to the aggregate downlink capacity of the largest MC-LAG group.

Hash Redistribution Modifying the LAG member set causes the hash function to remap some flows. This may produce transient packet reordering bounded by the depth of in-flight packets. Applications requiring strict ordering should employ higher-layer reordering tolerance.

Flap Dampening Implementations MUST support configurable dampening of member-set changes and relay state transitions to prevent rapid oscillation caused by link flapping.

Security Considerations The forwarding state preservation procedure () is a local forwarding-plane operation with no signaling-based attack surface. The peer notification via ICCP MUST be protected using the authentication mechanisms of . The ICL relay procedure () introduces relay messages that alter the forwarding topology. All relay messages MUST be authenticated. Implementations SHOULD support HMAC-SHA-256 keyed digests with pre-shared keys, consistent with . All secondary ICLs MUST traverse network segments under common administrative control. Implementations MUST support rate limiting of relay messages. The hop count ceiling provides an additional bound on message amplification. During failure conditions the ICL carries increased traffic. Operators SHOULD apply monitoring to detect unusual patterns that may indicate an attack.

IANA Considerations

ICCP Application Type IANA is requested to allocate one Application Type value for "MC-LAG ICL Relay" from the "ICC RG Parameter Type" registry established by . The reference is this document.

MC-LAG ICL Relay Message Types Registry IANA is requested to create a new registry, "MC-LAG ICL Relay Message Types", with an 8-bit value range (0-255). The assignment policy is "IETF Review" . The initial contents are below. Value 0 is Reserved. Initial MC-LAG ICL Relay Message Types

Value	Name	Reference
0	Reserved	This document
1	Failover-Request	This document
2	Failover-Response	This document
3	Liveness	This document
4	Revert-Request	This document
5	Revert-Response	This document
6-255	Unassigned

MC-LAG ICL Relay Status Codes Registry IANA is requested to create a new registry, "MC-LAG ICL Relay Failover-Response Status Codes", with an 8-bit value range (0-255). The assignment policy is "IETF Review" . The initial contents are the values defined in (0 = Success, 1 = No-Resource, 2 = Loop-Detected, 3 = Hop-Limit-Exceeded); values 4-255 are Unassigned.

References Normative References Informative References IEEE

Comparison with Existing Approaches

Failover Approach Comparison

Detailed Failover Timeline

Failover and Reversion Timeline G1 T1 m1 link-down {m2} X -> G1 T2 ICL added to G1 {m2, ICL} X -> G1 (unchanged) T3 Traffic via ICL to PE2 PE2 forwards No flooding via m3/m4 ... T10 m1 link-up {m1, m2, ICL} X -> G1 T11 ICL removed (hold timer) {m1, m2} X -> G1 (unchanged) ]]>