FANN Working Group J. Dong Internet-Draft Huawei Technologies Intended status: Informational 27 October 2026 Expires: 30 April 2027 Use Case and Requirements for Flow Control and Congestion Control Coordination in WAN-based AI DC Interconnect draft-dong-fann-fc-cc-coordinate-00 Abstract AI services are moving from single-site data center (DC) toward a multi-site distributed architecture. This requires to provide high- throughput, lossless, and deterministic network connectivity for AI workloads that span Data Center Networks (DCNs) over the Wide Area Network (WAN). Due to the long distance introduced by WAN, end-to- end congestion control across the DCN and WAN networks become a challenge, as the reaction time increases from microseonds to millisend level, which correspondingly increases the risk of packet loss both at the receiver side and in the network. This document describes the use case of coordinated flow control and congestion control to minimize the packet loss in WAN-based AI DCI networks. The requirements on Fast Network Notifications (FANN) mechanisms in this scenario is also described. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 30 April 2027. Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. Dong Expires 30 April 2027 [Page 1] Internet-Draft DCI CC Coordination October 2026 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Use Cases of Cordinated FC and CC . . . . . . . . . . . . . . 4 3.1. Congestion in WAN . . . . . . . . . . . . . . . . . . . . 4 3.2. Congestion in Egress DC . . . . . . . . . . . . . . . . . 7 4. Requirements on Fast Network Notifications . . . . . . . . . 10 5. Security Considerations . . . . . . . . . . . . . . . . . . . 11 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 11 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 11 8.1. Normative References . . . . . . . . . . . . . . . . . . 11 8.2. Informative References . . . . . . . . . . . . . . . . . 11 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 12 1. Introduction Driven by the need for larger model scales, higher compute density, and improved resource utilization, AI services are moving from single-site data center (DC) toward a multi-site distributed architecture. This requires to provide high-throughput, lossless, and deterministic network connectivity for AI workloads that span Data Center Networks (DCNs) over the Wide Area Network (WAN). Any packet loss or tail latency directly translates into stalled training pipelines, degraded job completion times (JCT), and wasted GPU cycles. A technical challenge arises from the fundamental difference between DCN and DCI scenarios is in congestion control and flow control. Within a single data center, the short physical distance keeps the Round-Trip Time (RTT) low, typically in the order of tens of microseconds, allowing the control loop to react quickly and maintain a stable, lossless fabric through mechanisms such as Explicit Congestion Notification (ECN), Congestion Notification Packet (CNP) and Priority Flow Control (PFC). In the distributed DCI scenario, the physical distance between DCs stretches the RTT to milliseconds or even tens of milliseconds. This significantly slows down the Dong Expires 30 April 2027 [Page 2] Internet-Draft DCI CC Coordination October 2026 control loop, making it difficult to promptly detect and react to congestion in the network. As a result, queue buildup and packet loss become far more likely, and the tight lossless guarantee achievable in DCN cannot be easily extended to the WAN domain. This motivates the need for coordinated congestion control and flow control across the WAN and the data center. [I-D.ietf-fann-problem-statement] describes the problem and limitations of existing approaches and documents the need for a fast notification mechanism. [I-D.song-fann-framework] provides the overall architecture, the functional role and information carried in fast network notifications. Rather than treating DC and WAN as independent network domains, AI DCI requires the congestion control and flow control mechanisms in both domains to work in an coordinated manner to preserve end-to-end lossless, high bandwidth utilization characteristics, so as to meet the performance demands of distributed AI workloads. This document describes the use case of coordinated congestion control and flow control in WAN based AI DCI to minimize the packet loss in both DCN and WAN networks. The requirements on a fast network notification mechanism in this scenraio is also described. 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 2. Terminology The following terminologies are used in this document: * PFC: Priority-based Flow Control * ECN: Explicit Congestion Notification * CNP: Congestion Notification Packet * RoCEv2: RDMA over Converged Ethernet Version 2 Dong Expires 30 April 2027 [Page 3] Internet-Draft DCI CC Coordination October 2026 3. Use Cases of Cordinated FC and CC Figure 1 shows a typical topology of distributed DC interconnect, in which two or more DCs are interconnected via the WAN network. The WAN network may be used only for AI DCI traffic, or it may also be used to carry other AI or non-AI services. Congestion may happen either within the DCs or in the WAN. The following of this section describes different congestion scenarios, and the coordination mechanisms in flow control and congestion control. +-------------------------+ +------------------------+ | Data Center A | | Data Center B | +----------+--------------+ +----------+-------------+ | | ------|---------------------------------|------- | | +---+ | | | | | R2|-----------------\ | | | | /+---+\ \ | | | | / \ \ | | | +-+-+/ +---+ \+-+-+ | | | R1 +---------+ R3+--------------+ R5| | | +---+\ +---+ /+---+ | | \ / / | | \+---+/ / | | | R4|-----------------/ | | +-|-+ DC Inter-connect Network | --------------|--------------------------------- | +----+--------------------+ | Data Center C | +-------------------------+ Figure 1. WAN-based AI DC Interconnect 3.1. Congestion in WAN Assume AI traffic is sent from DC-A (the ingress DC) to DC-B (the egress DC) taking the path R1-R3-R5 in WAN network. If a traffic burst or surge causes congestion on link R3-R5 (as shown in Figure 1), node R3 has the following options for handling congestion: Option 1: Receiver-Based Congestion Notification (Standard ECN) Dong Expires 30 April 2027 [Page 4] Internet-Draft DCI CC Coordination October 2026 R3 marks the ECN [RFC3168] field in the data packet as "Congestion Experienced (CE)" and forwards it to the receiver in the egress DC (DC-B). Upon detecting the CE mark, the receiver in DC-B generates a congestion notification message and sends it back to the traffic source in the ingress DC (DC-A), instructing the source to reduce its sending rate for the affected flows. This is the standard congestion notification mechanism used in DCN. However, in a WAN-based DC Interconnect scenario, the control loop spans the full WAN RTT, which is typically in the millisecond range. This latency may prevent timely rate reduction, leading R3 to drop packets due to buffer overflow. AI Traffic ------------------------------------------> +----------+ Congestion +----------+ | Data | +----+ +----+ +----+ | Data | | center A |---| R1 |---| R3 |-=-| R5 |---| center B | +----------+ +----+ +----+ +----+ +----------+ Packet with ECN Set |--------------------> Congestion Notification Message <-----------------------------------------| Figure 2. Receiver-Based Notification Option 2: Direct to Sender Congestion Notification R3 directly sends the congestion information to the traffic source in the ingress DC (DC-A). This approach is considered a "fast" congestion notification because it reduces the feedback path to approximately half-RTT. Upon receiving the notification, the sender can immediately reduce its sending rate for the identified flow. Although the feedback loop is reduced from one-RTT (Option 1) to half-RTT, the absolute latency in a WAN environment remains in the millisecond range. Consequently, the sending rate reduction may still not take effect quickly enough at the congestion node R3, and packet drops due to buffer overflow may still occur. Sending direct congestion notifications from WAN nodes presents two main challenges: * Packet Parsing: WAN nodes must parse data packets deeply to extract flow identification information. Dong Expires 30 April 2027 [Page 5] Internet-Draft DCI CC Coordination October 2026 * State Maintenance: For RoCEv2 flows, WAN nodes must obtain and maintain the mapping of source and destination Queue Pairs (QPs) in advance. AI Traffic ------------------------------------------> +----------+ Congestion +----------+ | Data | +----+ +----+ +----+ | Data | | center A |---| R1 |---| R3 |-=-| R5 |---| center B | +----------+ +----+ +----+ +----+ +----------+ <---------------------| Congestion Notification Message Figure 3. Direct to Sender Notification Option 3: Coordinated Flow Control and Congestion Control Notification Instead of sending the congestion information directly to the receiver in the egress DC or the source in the egress DC, R3 sends a WAN flow control notification to its upstream node R1 on the data path, notifying R1 to pause the specific flow which caused congestion. If R1 detects local congestion due to the pause of the flow, it can send a flow control notification to its upstream node on the data path. This hop-by-hop notification mechanism continues until the flow control signal reaches the gateway of the ingress DC (DC-A). The DC gateway pauses the flow according to the flow control notification. If this causes local congestion on the DC gateway, it may generate a congestion notification to the traffic source. For RoCEv2 flows, this requires the DC gateway to maintain the source and destination Queue Pair (QP) information. The advantage of this approach is, the control loop latency is comparable to that in Option 2, while it leverages hop-by-hop flow control within the WAN network. By pausing the traffic at each upstream node, the buffer resources on WAN nodes along the data path can absorb burst traffic before the source reduces its sending rate. This mechanism can effectively avoid or alleviate packet loss in the WAN network, even if the rate adjustment at the sender is delayed due to the long distance. Dong Expires 30 April 2027 [Page 6] Internet-Draft DCI CC Coordination October 2026 AI Traffic ------------------------------------------> +----------+ Congestion +----------+ | Data | +----+ +----+ +----+ | Data | | center A |---| R1 |---| R3 |-=-| R5 |---| center B | +----------+ +----+ +----+ +----+ +----------+ <-----|<-----|<-------| WAN WAN Congestion Flow Flow Notification Control Control Figure 4. Coordinated Flow Control and Congestion Notification 3.2. Congestion in Egress DC Assume traffic is sent from DC-A (the ingress DC) to DC-B (the egress DC) through the WAN network, taking the path R1-R3-R5. If a traffic burst or surge causes congestion within the egress DC (e.g., on link S1-L2 in Figure 5), the node detecting the congestion (node S1) has the following options for handling it: Option 1: Receiver-Based Congestion Notification (Standard ECN) S1 marks the ECN field in the data packet as "Congestion Experienced" and forwards it to the receiver in the egress DC (DC-B). Upon detecting the ECN mark, the receiver generates a congestion notification message and sends it back to the traffic source in DC-A, instructing the source to reduce its sending rate for the affected flows. This is the standard congestion notification mechanism used in DCN. However, in a DC interconnect scenario, the control loop spans the full WAN RTT, which is typically in the millisecond range. This latency may prevent timely rate reduction. Consequently, Priority- based Flow Control (PFC) may be used within DC-B to prevent S1 from dropping packets while waiting for the sending rate adjustment. If the burst is too large, S1 may still drop packets due to buffer overflow. Dong Expires 30 April 2027 [Page 7] Internet-Draft DCI CC Coordination October 2026 AI Traffic ------------------------------------------> DC-B +----------+ +------------+ | | +----+ +----+ +----+ | S1 S2 | | DC-A |---| R1 |---| R3 |---| R5 |---| / \ / \ | | | +----+ +----+ +----+ | L1 L2 L3 | +----------+ +------------+ Congestion Packet with ECN Set ---> Congestion Notification Message <---------------------------------------------| Figure 5. Receiver-Based Notification Option 2: Direct to Sender Notification S1 directly sends the congestion information to the traffic source in the ingress DC (DC-A). This approach is considered a "fast" congestion notification as it eliminates the delay of sending the notification to the receiver first. However, since the transmission time in the WAN is much larger than in the DC, the time for the source to receive the notification and for the rate reduction to take effect at the congestion node S1 is still at the WAN RTT (milliseconds) level. Thus, packet drops may still occur at S1 due to buffer overflow. Consequently, PFC may be used in DC-B to prevent S1 from dropping packets during this latency period. AI Traffic ------------------------------------------> DC-B +----------+ +------------+ | | +----+ +----+ +----+ | S1 S2 | | DC-A |---| R1 |---| R3 |---| R5 |---| / \/ \ | | | +----+ +----+ +----+ | L1 L2 L3 | +----------+ +------------+ Congestion Congestion Notification Message <------------------------------------------| Figure 6. Direct to Sender Notification Option 3: Coordinated Flow Control and Congestion Control Notification Dong Expires 30 April 2027 [Page 8] Internet-Draft DCI CC Coordination October 2026 This option introduces a coordinated flow control and congestion control mechanism across the DC and WAN networks. When congestion is detected at node S1 within DC-B, it initiates a PFC pause frame towards the DC-B Gateway. This step is critical for immediate packet loss prevention within the DC fabric. The DC-B Gateway receives the PFC pause frame and ensures that no packets are dropped on the S1-L2 link while the congestion signal is being processed and propagated. At the same time, the DC-B Gateway convert the congestion status into an fine-grained flow control signal and injects it into the WAN path at the first hop, R5. Unlike PFC, which is confined to Layer 2, the fine-grained flow control message can traverse the Layer-3 boundaries. Similar to the case described in section 3.1, option 3, the fine-grained flow control propagates hop-by-hop along the path, from R5 to R3, then to R1, and finally reaches the DC-A Gateway. Each WAN node receiving the flow control signal pauses the corresponding flow on its outgoing interface, which effectively throttles the traffic upstream before it reaches the congested point. Upon receiving the fine-grained flow control signal, the DC-A Gateway pauses the flow at the source edge. If this causes local congestion on the DC-A side, the gateway may trigger congestion notification message to the traffic source in DC-A for further rate reduction. For RoCEv2 flows, this requires the DC gateway to maintain source and destination Queue Pair (QP) mappings to ensure precise flow identification. The advantage of this coordinated approach is that, during the propagation of the congestion information to the source in DC-A, DC-B can be protected from packet loss, and the WAN links are protected from consequent congestion and buffer overflow, as reaction to the congestion can be performed at the intermediate WAN and DC nodes respectively. Dong Expires 30 April 2027 [Page 9] Internet-Draft DCI CC Coordination October 2026 AI Traffic ------------------------------------------> DC-B +----------+ +------------+ | | +----+ +----+ +----+ | S1 S2 | | DC-A |---| R1 |---| R3 |---| R5 |---| / \/ \ | | | +----+ +----+ +----+ | L1 L2 L3 | +----------+ +------------+ <-----|<-----|<-------|<----- |<------ |<----- WAN WAN WAN WAN DC Congestion Flow Flow Flow Flow Flow Notification Control Control Control Control Control Figure 7. Coordinated Flow Control and Congestion Notification 4. Requirements on Fast Network Notifications Section 3 demonstrates that in WAN-based DC interconnect scenarios, the coordination of flow control and congestion control between the WAN and the DC provides an effective approach to avoid packet loss due to congestion happened in either the WAN or the DCN. This coordination introduces new requirements for fast network notifications for flow control in WAN. The specific requirements are summarized as follows: * Fine Granularity: Flow control notifications in the WAN MUST NOT stop all traffic on a specific link or port. The pause signal SHOULD operate at a fine granularity, targeting specific flows, groups of flows, or traffic belonging to a specific tenant, etc. * Directed Delivery: Flow control notifications in the WAN MUST NOT be sent via broadcast or multicast. They SHOULD be delivered hop- by-hop along the reverse direction of the data path to upstream nodes, and MAY ultimately reach the ingress node of the specific traffic flows within the WAN. * Transport: Since flow control notifications need to be consumed by WAN network nodes in the data plane, it is RECOMMENDED that the notification message be carried within IP packets. * Coordination: Flow control notifications in the WAN MAY be generated either upon detecting congestion in the WAN network, or by converting flow control notifications received from downstream DCNs. Additionally, a DC gateway MAY use these notifications as a trigger to send a congestion notification to the source of the traffic flow causing the congestion. Dong Expires 30 April 2027 [Page 10] Internet-Draft DCI CC Coordination October 2026 5. Security Considerations The security considerations in [I-D.song-fann-framework] applies to this document. The Flow control notifications generated in WAN MUST be kept in the WAN network domain and MUST NOT be sent directly across the network boundary with DC networks. The fast flow control notification MAY contain nodes and path information of the WAN network, which MUST NOT be exposed to entities outside the WAN domain. 6. IANA Considerations This document does not require any IANA actions. 7. Acknowledgements The authors would like to thank Xuesong Geng and Xingjian He for their review and comments. 8. References 8.1. Normative References [I-D.song-fann-framework] Song, H., Dong, J., Ali, Z., Cheng, W., Zhu, Y., Pang, R., Contreras, L. M., and L. Guo, "A Framework for Fast Network Notifications", Work in Progress, Internet-Draft, draft-song-fann-framework-03, 26 October 2026, . [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, DOI 10.17487/RFC3168, September 2001, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . 8.2. Informative References Dong Expires 30 April 2027 [Page 11] Internet-Draft DCI CC Coordination October 2026 [I-D.ietf-fann-problem-statement] Dong, J., McBride, M., Clad, F., Zhang, Z. J., Zhu, Y., Xu, X., Zhuang, R., Pang, R., Lu, H., Liu, Y., Contreras, L. M., Mehmet, D., and R. Rahman, "Fast Network Notifications Problem Statement", Work in Progress, Internet-Draft, draft-ietf-fann-problem-statement-01, 26 October 2026, . Author's Address Jie Dong Huawei Technologies Huawei Campus, No.156 Beiqing Road Beijing, 100095 China Email: jie.dong@huawei.com Dong Expires 30 April 2027 [Page 12]