FANN Y. Liu Internet-Draft ZTE Corporation Intended status: Standards Track F. Qin Expires: 29 April 2027 China Mobile C. Gao ZTE Corporation 26 October 2026 Fast Network Notification for Plane Protection in Multi-Plane Networks draft-lqg-fann-multi-plane-protection-00 Abstract This document discusses the use of fast network notifications for plane evacuation in multi-plane networks. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 29 April 2027. Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Liu, et al. Expires 29 April 2027 [Page 1] Internet-Draft Fast Notification for Multi-Plane October 2026 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Multi-Plane Failure Scenarios . . . . . . . . . . . . . . . . 4 3.1. Scenario 1: NIC-to-Leaf Link Failure . . . . . . . . . . 4 3.2. Scenario 2: Leaf-to-Spine Link Failure . . . . . . . . . 5 3.3. Scenario 3: Plane Performance Degradation . . . . . . . . 7 3.4. Scenario Observations . . . . . . . . . . . . . . . . . . 7 4. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 7 4.1. Scope Requirements . . . . . . . . . . . . . . . . . . . 7 4.2. Consumer and Action Requirements . . . . . . . . . . . . 8 4.3. Delivery Path Requirements . . . . . . . . . . . . . . . 8 5. Security Considerations . . . . . . . . . . . . . . . . . . . 8 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 8 7.1. Normative References . . . . . . . . . . . . . . . . . . 8 7.2. Informative References . . . . . . . . . . . . . . . . . 8 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 9 1. Introduction Multi-plane networking is one of the common design choices in AI data centers. A high-capacity NIC is partitioned into multiple interfaces at port or sub-port granularity, and each interface is attached to an independent network fabric, referred to as a plane. [I-D.xu-idr-fare-in-mpson] describes such a topology for RoCE NICs in scale-out networks, and Multipath Reliable Connection (MRC) [MRC] uses multiple parallel planes with per-packet spraying and source- controlled path selection. When a plane becomes unreachable or degraded, the following methods are used to infer the failure: * Local detection at the NIC, e.g., link down or persistent link- level errors on the local port. * Active path health management, where the NIC continuously measures loss and latency across its paths using in-band measurements of data traffic and active probes, and steers traffic away from degraded paths, as done by MRC [MRC]. * Transport timeout or retransmission timeout, e.g., the RDMA timeout and retransmission mechanisms of RoCEv2 [IBTA-ROCEv2], which is end-to-end and RTT-bound. Liu, et al. Expires 29 April 2027 [Page 2] Internet-Draft Fast Notification for Multi-Plane October 2026 Local detection covers the NIC's own attachment well, but by construction does not extend to the remote side of a plane. Probing and transport timeouts observe the path end-to-end and therefore operate at RTT scale. Routing convergence detects and repairs reachability reliably, but completes on control-plane timescales. This document discusses the use of fast network notifications for plane protection in multi-plane networks. In-plane path switching or local repair performed by the switches within a plane is out of scope. 2. Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. Plane: An independent forwarding domain consisting of a set of network nodes and links, with no links to any other plane. Plane Attachment Node: A network node in a plane that is directly connected to one or more NIC interfaces. In a Clos-based deployment, a plane attachment node is a leaf switch. Plane-Internal Node: A network node in a plane that is not directly connected to any NIC interface. In a Clos-based deployment, these are spine or super-spine switches. In-Plane Repair: A forwarding or path adjustment performed inside a plane by the nodes of that plane. Plane Evacuation: The action taken by a source NIC to move traffic away from an unreachable or degraded plane to one or more other planes. Plane Exclusion: The state in which a plane is removed from the candidate plane set for a given destination. Source NIC: The NIC that originates traffic and performs plane selection for that traffic. Destination NIC: The NIC that receives traffic in a given flow. Liu, et al. Expires 29 April 2027 [Page 3] Internet-Draft Fast Notification for Multi-Plane October 2026 3. Multi-Plane Failure Scenarios This section describes two representative failure scenarios in a multi-plane network. Plane 1 Plane 2 Plane 3 +---------------+ +---------------+ +---------------+ | +--+ | | +--+ | | +--+ | | |S1| | | |S1| | | |S1| | | +--+ | | +--+ | | +--+ | | / \ | | / \ | | / \ | | +--+ +--+ | | +--+ +--+ | | +--+ +--+ | | |L1| |L2| | | |L1| |L2| | | |L1| |L2| | | +--+--+--+ + | +--+--+--+ + | +--+--+--+--+ +-------|-------+ +-------|-------+ +-------|-------+ | | | | | | | | | +-------|-------------------|-------------------|--------+ | NIC1 | +--------------------------------------------------------+ Figure 1: Example Multi-Plane Topology Figure 1 illustrates an example multi-plane topology in a data center. Each plane is a two-tier Clos fabric with spine and leaf switches. A NIC is attached to a leaf switch in every plane. The figure shows three planes for illustration. 3.1. Scenario 1: NIC-to-Leaf Link Failure Figure 2 shows a single plane with two leaf switches and two spine switches. Leaf1 is attached to NIC N1 and N2. Leaf2 is attached to NIC N3 and N4. Leaf and spine switches are fully meshed. Liu, et al. Expires 29 April 2027 [Page 4] Internet-Draft Fast Notification for Multi-Plane October 2026 +------+ +------+ |Spine1| |Spine2| +--+---+ +---+--+ | \ / | | \ / | | \ / | | X | | / \ | | / \ | +--+---+ +---+--+ |Leaf1 | |Leaf2 | +-+--+-+ +--+-+-+ | | | | +------+ +---+ + +----+ | | | | +-+--+ +-+--+ +-+--+ +-+--+ | N1 | | N2 | | N3 | | N4 | +----+ +----+ +----+ +----+ Figure 2: NIC-to-Leaf Link Failure in a Plane In Plane 1, the link between Leaf2 and N4 fails. As a result, other NICs in Plane 1 cannot reach N4 through this plane. Within this plane, N4 is reachable only through Leaf2. Therefore, an in-plane forwarding adjustment, such as Leaf2 notifying a spine to change its path, cannot solve the problem: the spine can still reach Leaf2, but Leaf2 has lost its link to N4. The source NICs need to be informed so that they can evacuate traffic destined to N4 from this plane. 3.2. Scenario 2: Leaf-to-Spine Link Failure Figure 3 shows a single plane with three leaf switches and two spine switches. Leaf1 is attached to N1 and N2. Leaf2 is attached to N3 and N4. Leaf3 is attached to N5 and N6. Leaf and spine switches are fully meshed. Liu, et al. Expires 29 April 2027 [Page 5] Internet-Draft Fast Notification for Multi-Plane October 2026 +------+ +------+ |Spine1| |Spine2| +--+---+ +---+--+ | | | \ / | | \ / | | \ / | | \ / | | \ / | | \ / | | \ / | | X | | / \ | | / \ | | / \ | | / \ | | / \ | | / \ | | / \ | +----+----+ +-------+ +--+-----+ | Leaf1 | | Leaf2 | | Leaf3 | +----+----+ +---+---+ +--+-----+ | | | +---------+----+ +----+----+ +----+---------+ | | | | | | +-+--+ +-+--+ +-+--+ +-+--+ +-+--+ +-+--+ | N1 | | N2 | | N3 | | N4 | | N5 | | N6 | +----+ +----+ +----+ +----+ +----+ +----+ Figure 3: Leaf-to-Spine Link Failure in a Plane In Plane 1, all links between Leaf1 and the spine switches fail. N1 and N2 can still communicate with each other through Leaf1, but N3 through N6 cannot reach N1 or N2 through this plane, and N1 and N2 cannot reach N3 through N6 through this plane. The failure is bidirectional: traffic from N1/N2 to remote NICs and traffic from remote NICs to N1/N2 are both affected. The detection points are Leaf1 and the spines, but Leaf1 can no longer send any packet to remote nodes within this plane, and remote nodes do not know that N1/ N2 are unreachable through this plane. Liu, et al. Expires 29 April 2027 [Page 6] Internet-Draft Fast Notification for Multi-Plane October 2026 3.3. Scenario 3: Plane Performance Degradation In addition to hard failures, the source NIC may need to be informed about performance degradation at the plane granularity, where the plane is still reachable, but it is no longer optimal for carrying the affected traffic. For example, a link inside the plane may experience severe congestion or microbursts, causing high latency and packet loss, while the paths remain physically connected. The topology in this case is the same as in Figure 2 or Figure 3. However, unlike Scenario 1 and Scenario 2 where the links are physically down, here the links remain operationally up, but the available capacity is reduced or the path quality is degraded. 3.4. Scenario Observations The failure scenarios share the following characteristics: * The failure is scoped to a single plane, in-plane forwarding adjustment cannot resolve the failure. * The impact of a single failure is set-level and bidirectional: the failure affects a set of NICs attached to the same leaf switch, and it affects both directions: the NICs in the set cannot reach remote NICs through the plane, and remote NICs cannot reach the set through the plane. * Detection and mitigation are performed by different nodes, and the detector cannot always reach the nodes that need to act. 4. Requirements In addition to the considerations already described in the FANN documents [I-D.ietf-fann-problem-statement], the multi-plane evacuation scenario imposes the following requirements. 4.1. Scope Requirements *R-S1:* A consumer MUST be able to determine, from the notification message, which plane the failure occurred in. *R-S2:* A consumer MUST be able to distinguish, from the notification message, whether the message is a plane failure notification or an ordinary link failure notification. *R-S3:* A plane-scoped notification MUST NOT leak across planes. When propagated across planes, the content MUST preserve the semantics of "about one plane only". Liu, et al. Expires 29 April 2027 [Page 7] Internet-Draft Fast Notification for Multi-Plane October 2026 *R-S4:* In addition to plane failure notifications, plane degradation notifications MAY be supported. In this case, a consumer MUST be able to distinguish three states: plane unreachable, plane degraded, and destination unreachable within a plane. *R-S5:* The information elements of the notification MUST be sufficient to drive plane-granular exclusion or quarantine, rather than only path-granular adjustment. 4.2. Consumer and Action Requirements *R-C1:* The consumers of the notification MUST include at least the source NIC of the affected flow. 4.3. Delivery Path Requirements *R-D1:* The delivery path of the notification MUST NOT share fate with the failed link or the failed plane uplink. *R-D2:* The solution MUST explicitly address the case where the in- plane path from the detection point to some consumers has been cut. For example, the notification may be delivered through another plane or an out-of-band channel, or the affected consumers may be required to rely on other mechanisms such as local end-to- end probing or routing convergence. 5. Security Considerations TBD. 6. IANA Considerations This document has no IANA actions. 7. References 7.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . 7.2. Informative References Liu, et al. Expires 29 April 2027 [Page 8] Internet-Draft Fast Notification for Multi-Plane October 2026 [I-D.ietf-fann-problem-statement] Dong, J., Ed., McBride, M., Ed., and F. Clad, Ed., "Fast Network Notifications Problem Statement", Work in Progress, Internet-Draft, draft-ietf-fann-problem- statement-00, 27 August 2026, . [I-D.xu-idr-fare-in-mpson] Xu, X., Patel, K., Wang, H., Sun, W., and G. Ma, "Fully Adaptive Routing Ethernet in Multi-Plane Scale-Out Networks", Work in Progress, Internet-Draft, draft-xu-idr- fare-in-mpson-01, 16 August 2026, . [IBTA-ROCEv2] InfiniBand Trade Association, "InfiniBand Architecture Specification Volume 1, Release 1.4", 2020, . [MRC] Araujo, J., Chow, A., Handley, M., Lu, G., and L. Yuan, "Resilient AI Supercomputer Networking using MRC and SRv6", May 2026, . Authors' Addresses Yao Liu ZTE Corporation Email: liu.yao71@zte.com.cn Fengwei Qin China Mobile Email: qinfengwei@chinamobile.com Chenqiang Gao ZTE Corporation Email: gao.chenqiang@zte.com.cn Liu, et al. Expires 29 April 2027 [Page 9]