China Mobile

Beijing100053 China lizhiqiangyjy@chinamobile.com

China Mobile

Beijing100053 China duzongpeng@chinamobile.com

Centec

Shanghai201203 China wangjj@centec.com

Centec

Shanghai201203 China chengw@centec.com

Centec

Shanghai200000 China zhaoying@centec.com

SAIA

Shanghai200125 China chunhao.zhao@sh-aia.com

Inesa

Shanghai200030 China sunxun@inesa.com

Web and Internet Transport incast receiver-driven credit congestion control all-to-all distributed inference limited domain Expert-parallel and disaggregated (prefill/decode) inference deployments generate many-to-one ("incast") traffic in which many senders transmit to one receiver at the same time. Mitigations based on priority queues, enlarged switch buffers, and disabled Explicit Congestion Notification (ECN) marking stop scaling when the bandwidth-delay product of the network grows. This document specifies Receiver-Driven Credit Control (RDCC), a UDP-based signaling protocol with which a receiver bounds the volume of data that all senders may have in flight toward it. RDCC defines a receiver credit budget, a bounded unscheduled first window for short messages, cumulative per-message and class-weighted credit grants, sender backlog indications with suggested credit rates, and optional congestion reports from on-path nodes. RDCC operates within a limited domain, complements the congestion control of the data transport, and does not change the format of data packets.

Introduction

Problem Statement In Mixture-of-Experts (MoE) inference with Expert Parallelism (EP), the dispatch and combine phases of every layer perform an all-to-all exchange among the expert-hosting accelerators. In prefill/decode disaggregated serving, the key-value (KV) cache produced by prefill instances is moved to decode instances. Both patterns frequently make several senders target the same receiver within a few microseconds. For example, when four accelerators on each of two servers send to one accelerator on a third server, eight flows converge on one receiver port (an 8-to-1 incast). Collective library optimizations that aggregate traffic inside a server reduce the fan-in but do not remove it . Deployed mitigations fall into three groups: (1) placing all-to-all traffic into a dedicated high-priority queue; (2) reserving more buffer and a larger bandwidth share for that queue in network interface cards (NICs) and switches; and (3) disabling ECN marking on that queue so that rate-based congestion control such as DCQCN does not react to micro-bursts. The buffer needed by (2) grows with the bandwidth-delay product (BDP). Across long-distance or multi-tier fabrics the required buffer exceeds what switches provide, and (3) then converts congestion into loss or, with Priority-based Flow Control (PFC) , into head-of-line blocking that spreads beyond the congested port. A large standing buffer also adds queueing delay to every message, which directly lengthens the time per output token of the inference service. The receiver is the only point that observes all of the demand converging on its access link. A receiver-driven scheme lets that receiver admit data at the rate its link and buffers can absorb, before the data enters the network. Receiver-driven transports have been explored in research systems such as NDP and Homa , and in industry specifications , but no IETF protocol provides an interoperable way for independently implemented senders, receivers, and network nodes to exchange credit and congestion information for this traffic.

Protocol Overview RDCC separates admission from transmission. Data continues to be carried by an existing message transport, for example RDMA over Converged Ethernet version 2 (RoCEv2) or a UDP-based message transport. RDCC adds a control channel, carried in UDP, over which the receiver tells each sender how many octets of each message it may transmit and at what rate. The main elements are:

• a receiver credit budget G that bounds the octets granted but not yet received across all senders ();
• a bounded unscheduled first window that lets messages no larger than the Unscheduled Limit U start without a round trip, spread by a random start delay ();
• cumulative, idempotent credit grants per message or per sender pool, with an explicit lifetime ();
• class-weighted division of pool credit and the associated Differentiated Services marking ();
• sender backlog indications and suggested credit rates, which let a sender that receives credit from many receivers steer the receivers toward its own bottleneck rate (); and
• optional congestion reports from on-path nodes whose queue exceeds a threshold ().

Scope and Applicability RDCC is designed for limited domains such as an inference data center or a set of data centers under a single administration, connected by a network whose configuration (Differentiated Services Code Points, ECN, queue thresholds) is known to the operator. RDCC MUST NOT be used across the general Internet. RDCC does not replace the congestion control of the data transport. A sender transmits no faster than the smaller of the rate allowed by its transport congestion control and the rate granted by RDCC (). RDCC does not define data packet formats, does not require changes to switch forwarding hardware, and does not depend on PFC. Queue configuration, credit allocation algorithms beyond the invariants of this document, and the means by which applications divide traffic into messages are local matters.

Relationship to Other Work describes transport considerations for distributed inference networks and defines no protocol elements; RDCC is one protocol that addresses the incast consideration identified there. reserves credit hop by hop with RSVP for RDMA across a wide-area network; RDCC is end-to-end, signals per message, and needs no per-flow state in routers. The Dynamic Link Exchange Protocol credit extensions govern a single radio link between a router and a modem. allocates transmission authority from a global ledger for training fabrics, whereas RDCC keeps authority at each receiver. Fast network event notification for AI fabrics is being studied in ; the on-path report of could be carried by such a mechanism once one is specified.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Message:

An application data unit of known length that the sender hands to the data transport as one unit, identified within an association by a 64-bit Message ID.

Sender, Receiver:

The endpoints that transmit and receive messages. A host can be both.

Association:

The RDCC state shared by one sender and one receiver, identified by an Association ID chosen by the receiver.

Credit:

Permission, issued by a receiver, to transmit a number of octets. Credit is expressed as a cumulative octet offset.

Credit Budget (G):

The maximum number of octets that a receiver has granted, across all associations, and not yet received or reclaimed.

Unscheduled Limit (U):

The number of octets of a message that a sender may transmit before receiving credit for it.

Start Jitter Limit (J):

The upper bound of the random delay that a sender applies before an unscheduled transmission.

Backlog Threshold (N_cu):

The amount of granted but untransmitted credit above which a sender reports its own congestion.

Queue Threshold (B_th):

The queue occupancy above which a Congestion Reporting Node emits reports.

Congestion Reporting Node (CRN):

An on-path switch or router that implements .

Traffic Class:

One of eight sender-local priority classes (0..7) that share the credit of a pool grant according to weights.

All multi-octet fields are transmitted in network byte order. Rates are expressed in megabits per second (Mbit/s) and durations in microseconds unless stated otherwise.

Architecture shows the roles. Control messages travel in UDP between sender and receiver. A CRN observes data packets in its queues and, when a queue exceeds B_th, sends a report either toward the receiver (for credit-scheduled traffic) or toward the sender (for unscheduled traffic).

RDCC Roles and Signaling Paths +----------+ | Sender 1 |---+ data (existing transport) +----------+ | +-------+ +----------+ +---->| |-------->| | +----------+ | CRN | | Receiver | | Sender 2 |-------->| |-------->| | +----------+ +-------+ +----------+ RDCC control messages (UDP, RDCC port): Sender -> Receiver : ASSOC_REQUEST, CREDIT_REQUEST, SENDER_STATUS, CREDIT_RETURN, CANCEL Receiver -> Sender : ASSOC_ACCEPT, CREDIT_GRANT, MSG_DONE, CANCEL, ERROR CRN -> Receiver : CONGESTION_REPORT (scheduled traffic) CRN -> Sender : CONGESTION_REPORT (unscheduled traffic)

A receiver keeps, for each association, the cumulative credit it has granted for each message and the number of octets it has received. A sender keeps, for each message, the cumulative credit it holds and the number of octets it has transmitted. Every grant restates a cumulative offset, and stale or duplicated control messages are discarded by sequence number (), so loss, duplication, or reordering of control messages never causes a sender to exceed the credit that the receiver intended to issue. The receiver relates arriving data to messages through a transport binding (): each CREDIT_REQUEST names the transport-level identity of the message, so that the receiver can count the octets received for it from the information its transport already provides, without any change to data packets.

Protocol Operation

Association Establishment Before sending messages under RDCC, a sender sends ASSOC_REQUEST to the receiver's RDCC port, carrying a 64-bit random Sender Nonce, its capabilities, and its line rate. The receiver answers with ASSOC_ACCEPT, which echoes the nonce and assigns a non-zero Association ID and the receiver's current Epoch. ASSOC_ACCEPT also carries the operating parameters of the association: U, J, N_cu, the default Grant Lifetime, and the Credit Unit. A receiver that cannot admit a new association answers with ERROR (reason ASSOC_REFUSED). The sender retransmits ASSOC_REQUEST with exponential backoff, starting at 1 ms and doubling, for at most eight attempts; if none is answered, the sender concludes that the receiver does not support RDCC and sends without it. A receiver MUST treat a retransmitted ASSOC_REQUEST with the same nonce as idempotent and return the same Association ID. The Sender Nonce MUST be generated by a cryptographically secure random number generator; it binds the response to the request and prevents off-path injection of ASSOC_ACCEPT. The receiver sends all messages of the association to the address and port from which the accepted ASSOC_REQUEST came. A receiver changes the parameters of an existing association by sending ASSOC_ACCEPT with the Parameter Update flag. The sender acknowledges it by sending ASSOC_REQUEST with the same Sender Nonce and the Update Acknowledgment flag; the receiver retransmits the update with exponential backoff until it is acknowledged. Until then, the sender may still act on the old parameters, so the receiver accounts for it with the larger of the old and new U. The Epoch identifies an incarnation of the receiver's RDCC state. A receiver that loses its state (for example, after a restart) MUST use a different Epoch. A sender accepts a new Epoch only from an ASSOC_ACCEPT that echoes its current Sender Nonce. A sender that receives ERROR (UNKNOWN_ASSOCIATION) carrying its Association ID MUST discard all credit held for that association, re-establish it, and request credit again for its pending messages. Messages from CRNs carry no Epoch and do not affect associations. Associations are kept alive by data and control traffic. A receiver MAY remove an association that has carried no traffic for a locally configured idle time, which SHOULD be at least 10 seconds; it reclaims all credit of that association when doing so.

Transmission Mode Selection For each new message the sender selects one of two modes:

Unscheduled mode:

If the message length is less than or equal to U, the sender MAY transmit the whole message without waiting for credit, subject to .

Scheduled mode:

Otherwise, the sender MUST send CREDIT_REQUEST and MUST NOT transmit any octet of the message until it holds credit for that octet.

A sender MAY use scheduled mode for any message. U is advertised per association, so the receiver alone decides how much unscheduled data each sender may inject. A receiver that advertises U = 0 disables unscheduled mode for that association.

Unscheduled Transmission A sender using unscheduled mode:

1. MUST delay the first transmission by a random duration drawn uniformly from [0, J] microseconds, so that senders reacting to the same application event do not start simultaneously;
2. MUST send a CREDIT_REQUEST for the message no later than the first data packet, with the Unscheduled Sent field set to the message length, so that the receiver can account for the octets;
3. MUST NOT have more than one unscheduled message outstanding on an association; an unscheduled message stops being outstanding when the sender receives MSG_DONE or a CREDIT_GRANT for it;
4. MUST mark the data packets with the Differentiated Services Code Point (DSCP) that the domain configures for unscheduled RDCC traffic, if one is configured; and
5. MUST stop unscheduled transmission and convert the remainder of the message to scheduled mode upon receiving a CONGESTION_REPORT addressed to it, or a link-layer pause indication if the local interface supports PFC.

The receiver accounts each unscheduled message against a portion G_u of its budget (). The receiver MUST choose U for each association so that the sum of U over all associations with a non-zero U does not exceed G_u; because each association can have only one unscheduled message outstanding, this bounds all unscheduled data in flight. A sender that converts an unscheduled message to scheduled mode sends a new CREDIT_REQUEST whose Unscheduled Sent field states the octets actually transmitted. The receiver releases the unscheduled reservation of a message when the announced octets have arrived, or when J plus the default Grant Lifetime plus 2 * D_max () has elapsed after it received the CREDIT_REQUEST.

Credit Requests and Grants CREDIT_REQUEST announces a message: its Message ID, total length, the number of octets already sent in unscheduled mode, its Traffic Class and weight, a Service Hint, its transport binding, and an optional deadline. Message IDs MUST increase with each new message of an association and MUST NOT be reused within it. The sender retransmits CREDIT_REQUEST if it holds no credit for the message after a request timeout, initially twice the smoothed round-trip time and at least 100 microseconds, doubling on each retransmission up to 100 ms. CREDIT_GRANT carries a Grant Limit, the cumulative octet offset (exclusive) up to which the sender may transmit, a Pacing Rate, and a Lifetime. The following rules apply:

1. A grant with a Grant Limit that is not larger than the limit the sender already holds for that message only refreshes the Lifetime and Pacing Rate; it never reduces credit.
2. The sender MUST NOT transmit an octet at offset o of a message unless o is smaller than the Grant Limit held for that message, or the octet is part of the unscheduled window permitted by .
3. The sender MUST NOT transmit under a grant after the Lifetime of that grant has elapsed, measured from its reception. It sends a new CREDIT_REQUEST to obtain fresh credit.
4. The sender MUST NOT transmit the message faster than the Pacing Rate, averaged over any interval equal to the transmission time of ten Credit Units at that rate. Receivers MUST issue a non-zero Pacing Rate.
5. The receiver MUST NOT grant beyond the message length. It sets the FINAL flag in the grant that reaches the message length.

A grant with Message ID zero and the POOL flag set is a pool grant: it covers all scheduled messages of the association, and the Grant Limit counts octets across those messages. Pool grants allow a sender to divide credit among its messages according to without a round trip per message. A receiver MUST NOT use pool grants with a sender that did not advertise the POOL capability. When the receiver has received all octets of a message it sends MSG_DONE. The sender MAY abandon a message by sending CANCEL; the receiver then reclaims the outstanding credit of the message after the reclamation delay of . A sender that will not use credit it holds SHOULD return it with CREDIT_RETURN, which states the offset beyond which the sender will not transmit.

Receiver Credit Budget For each association a, let Granted(a) be the sum, over the scheduled messages of a, of the Grant Limit issued minus the octets received, and let Unsched(a) be the unscheduled octets announced but not yet received. A receiver MUST maintain: sum over a of ( Granted(a) + Unsched(a) ) <= G G is divided into a scheduled part G_s and an unscheduled part G_u, with G = G_s + G_u. To keep the receiver's link busy, G SHOULD be at least the BDP of the receiver's access link, computed from the link capacity and the base round-trip time of the domain, and G MUST NOT exceed the buffer the receiver can commit to arriving data. The values of G, G_s, and G_u are configured or derived locally. Credit is released when data arrives, when MSG_DONE is sent, or when credit is reclaimed. Let D_max be the maximum one-way delay of the domain. A sender measures the Lifetime from the reception of the grant, which can be up to D_max after the receiver sent it, and an octet transmitted just before expiry can take another D_max to arrive. The receiver therefore reclaims the credit of an expired grant no earlier than Lifetime + 2 * D_max after it sent the grant, and the credit of a cancelled message no earlier than 2 * D_max after it sent or received the CANCEL.

Credit Allocation The receiver decides which messages receive credit and at what rate. The choice of algorithm is local, provided that the invariant of holds and the following requirements are met:

1. The sum of the Pacing Rates the receiver issues MUST NOT exceed the capacity of its access link.
2. The receiver MUST reduce the rate it grants to a sender when it observes congestion on the path from that sender: ECN Congestion Experienced (CE) marks on data from that sender, a CONGESTION_REPORT naming a flow from that sender, or a round-trip time above the configured target.
3. When the most recent SENDER_STATUS from a sender carries a non-zero Suggested Rate, the Pacing Rate granted to that sender MUST NOT exceed it. The receiver grants the smaller of the Suggested Rate and the rate that its remaining budget allows.
4. The receiver MUST NOT grant credit to an association whose Epoch or Association ID it does not recognize.

An additive-increase multiplicative-decrease (AIMD) controller is RECOMMENDED as the default. Every update interval T_upd (by default the base round-trip time) the receiver adjusts the rate r_i of each active sender i: if congestion signal s_i > 0 in the last interval: r_i = max(R_min, r_i * (1 - beta * s_i)) else: r_i = min(R_max_i, r_i + alpha) where s_i in [0, 1] is the largest of the CE fraction, the normalized Congestion Degree (), and the normalized excess delay; beta defaults to 0.5; alpha defaults to one percent of the access link capacity; R_min is one Credit Unit per update interval; and R_max_i is the smaller of the sender's line rate and its Suggested Rate. Rate-based controllers in the style of BBR MAY be used instead. Credit that becomes available is distributed among waiting messages by Traffic Class first (lower class number first) and by weight within a class. A receiver SHOULD give precedence to messages with an earlier deadline when deadlines are present.

Traffic Classes and Pool Credit A sender often carries several flows to the same receiver, such as dispatch traffic, KV cache transfers, and background replication. With pool grants, the sender divides pool credit among its scheduled messages as follows: classes are served in strict priority order from 0 to 7, and within a class credit is divided in proportion to the weights carried in CREDIT_REQUEST, using a work-conserving weighted round-robin. The sender MUST mark the data packets of each class with the DSCP that the domain configures for that class; the mapping from Traffic Class to DSCP is a domain-wide configuration that SHOULD follow . A receiver that uses per-message grants applies the same precedence when choosing which messages to serve.

Sender Backlog Indication and Suggested Rates A sender that receives credit from many receivers can become the bottleneck itself: its own link or NIC cannot use all the credit it holds. Credit held idle at such a sender is wasted capacity at the receivers that issued it. A sender's Backlog Threshold is the smallest N_cu advertised by its receivers, or a smaller locally configured value. When the total credit the sender holds and has not yet used, summed over all its associations, exceeds its Backlog Threshold, the sender MUST send SENDER_STATUS on each association that holds such credit. SENDER_STATUS carries the Unused Credit on that association, a Congestion Level from 0 (none) to 255 (saturated), and a Suggested Rate. The sender SHOULD compute the Suggested Rate for receiver j as its own available transmission rate multiplied by the share it wishes receiver j to obtain, where the shares are derived from the Service Hints of the pending messages (for example, larger shares for latency-sensitive messages) and sum to at most one. The sender repeats SENDER_STATUS once per smoothed round-trip time while the condition persists, and sends one final SENDER_STATUS with Congestion Level zero when it ends. A receiver ignores a Suggested Rate from a SENDER_STATUS received more than four default Grant Lifetimes ago, so a lost final SENDER_STATUS cannot cap a sender indefinitely. Receivers apply the third rule of . Because every receiver of a congested sender learns its intended share directly, the receivers converge to the sender's allocation within one round trip instead of discovering it through loss or delay.

On-Path Congestion Reporting Support for this section is OPTIONAL for network nodes. Receivers and senders MUST be able to process CONGESTION_REPORT. A CRN monitors the occupancy Q of each egress queue that carries RDCC traffic. While Q exceeds B_th, the CRN MAY generate CONGESTION_REPORT for packets dequeued from that queue, subject to a rate limit of at most one report per flow per configured report interval (RECOMMENDED: 50 microseconds) and an aggregate limit per port. The report identifies the data flow by its addresses, protocol, and ports and carries:

• the Congestion Degree, computed as min(255, floor(255 * (Q - B_th) / (Q_max - B_th))), where Q_max is the queue capacity;
• the queue occupancy and threshold, in octets; and
• optionally, a Suggested Rate equal to the queue's service rate divided by the number of flows the CRN observes in the queue.

The CRN classifies the packet by its DSCP. For scheduled traffic it sends the report to the destination address of the data packet (the receiver), so that the receiver adjusts its grants. For unscheduled traffic it sends the report to the source address (the sender), so that the sender stops its unscheduled transmission (). Reports are sent to the RDCC UDP port with the Association ID and Epoch fields set to zero. Where in-situ OAM is deployed in the domain, a receiver MAY derive the same congestion information from the queue depth data field, and where ECN is enabled, from CE marks. These signals and CONGESTION_REPORT are equivalent inputs to . A CONGESTION_REPORT can only cause a receiver to reduce the rate it grants, or a sender to stop unscheduled transmission. An endpoint MUST NOT increase any rate or credit in response to a CONGESTION_REPORT. An endpoint MUST ignore reports whose source address is not within the set of CRN addresses configured for the domain.

Transport Bindings The Binding Kind and Binding Value of CREDIT_REQUEST identify the message at the data transport. This document defines:

Binding Kind 1, RoCEv2:

An RDMA interface cannot pause in the middle of an RDMA message, so a sender that honors partial grants divides an RDCC message into several RDMA messages and posts each one only when it holds credit for all of its octets. The Binding Value carries, in bits 63 to 40, the destination queue pair number; in bits 39 to 16, the message sequence number that the responder will assign to the first of these RDMA messages ; and in bits 15 to 0, the number of RDMA messages. Because messages on a queue pair are processed in order, the sender knows these numbers when it posts the messages. The receiver MUST be able to count the payload octets received on the queue pair, for example with per-queue-pair receive counters of its RDMA interface, since RDMA WRITE operations produce no completion at the responder.

Binding Kind 2, Opaque:

The Binding Value is a 64-bit message identifier that the data transport carries in its own headers, with a meaning agreed by configuration.

A receiver that does not support the Binding Kind of a request answers with ERROR (UNSUPPORTED_BINDING) for that message.

Error Handling and Recovery

• A receiver that receives a CREDIT_REQUEST for an unknown association replies with ERROR (UNKNOWN_ASSOCIATION); the sender re-establishes the association and re-requests credit for its pending messages.
• Data received for a message beyond its Grant Limit and outside its unscheduled window indicates a misbehaving sender. The receiver MAY discard such data, MUST NOT grant further credit to the association for that message, and SHOULD log the event.
• If a receiver is about to run out of budget for an extended period, it MAY send ERROR (BUDGET_EXHAUSTED) to senders with pending requests so that they can choose other destinations; senders keep their requests outstanding.
• Messages carrying an unknown Version MUST be discarded. Messages with an unknown Type MUST be discarded; if the message belongs to an established association, the receiver MAY reply with ERROR (UNSUPPORTED_TYPE) to the address bound to that association, and it MUST NOT reply otherwise.
• A message whose Length field is smaller than the size defined for its Type, or larger than the UDP payload, MUST be discarded.

Message Formats RDCC messages are carried in UDP to or from the RDCC port (). One UDP datagram carries exactly one RDCC message. Senders MUST set the UDP checksum.

Port Configuration The RDCC port is a UDP port configured by the operator of the limited domain; this document does not request a port assignment from the "Service Name and Transport Protocol Port Number Registry" . All senders, receivers, and CRNs of a domain MUST be configured with the same RDCC port, because CRNs send reports to endpoints without prior signaling. Operators SHOULD choose the port from the Dynamic Ports range (49152-65535) and MUST ensure that no other service of the domain uses it. Implementations MUST NOT assume a default port value.

Common Header

RDCC Common Header 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Version| Rsv | Type | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Association ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Epoch | Flags | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Version (4 bits):

1.

Rsv (4 bits):

Set to zero on transmission and ignored on receipt.

Type (8 bits):

Message type ().

Length (16 bits):

Length of the message in octets, including this header.

Association ID (32 bits):

Assigned by the receiver in ASSOC_ACCEPT; zero in ASSOC_REQUEST and in messages from CRNs.

Epoch (16 bits):

The receiver's incarnation (); zero in ASSOC_REQUEST and in messages from CRNs.

Flags (16 bits):

Type-specific flags. Undefined flags are set to zero and ignored on receipt.

Sequence Number (32 bits):

Incremented by one for each message the originator sends on the association (or, for a CRN, per reporting port), including retransmissions. An endpoint MUST discard a message of an association whose Sequence Number is not newer, in serial number arithmetic , than the newest one it has already processed from the same originator on that association. ASSOC_REQUEST, ASSOC_ACCEPT that echoes the sender's current nonce, ERROR with reason UNKNOWN_ASSOCIATION, and messages from CRNs are exempt, because a receiver that has lost its state also restarts its sequence numbers; a sender that accepts a new Epoch resets the newest processed Sequence Number of the association.

RDCC Message Types

Type	Name	Direction	Length
0x01	ASSOC_REQUEST	S to R	48
0x02	ASSOC_ACCEPT	R to S	48
0x03	CREDIT_REQUEST	S to R	56
0x04	CREDIT_GRANT	R to S	40
0x05	SENDER_STATUS	S to R	32
0x06	CONGESTION_REPORT	CRN to R or S	48 or 72
0x07	MSG_DONE	R to S	32
0x08	CANCEL	S to R or R to S	28
0x09	CREDIT_RETURN	S to R	32
0x0A	ERROR	any	28

ASSOC_REQUEST and ASSOC_ACCEPT

ASSOC_REQUEST (Type 0x01) and ASSOC_ACCEPT (Type 0x02) Body 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sender Nonce (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Capabilities | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Line Rate / Unscheduled Limit U | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved / Start Jitter J | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved / Backlog Threshold N_cu | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved / Grant Lifetime | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved / Credit Unit | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

In ASSOC_REQUEST, the fields after Capabilities carry the sender's Line Rate (Mbit/s) followed by four reserved words set to zero. The reserved words make ASSOC_REQUEST as large as ASSOC_ACCEPT, so the exchange cannot be used for amplification. In ASSOC_ACCEPT, the fields carry U (octets), J (microseconds), N_cu (octets), the default Grant Lifetime (microseconds), and the Credit Unit (octets, typically the path MTU payload). Capability bits: 0x00000001 POOL (pool grants supported), 0x00000002 STATUS (SENDER_STATUS supported), 0x00000004 CRN (on-path reports are processed). Flag 0x0001 in ASSOC_ACCEPT is Parameter Update: the message updates the parameters of an existing association. Flag 0x0002 in ASSOC_REQUEST is Update Acknowledgment ().

CREDIT_REQUEST

CREDIT_REQUEST (Type 0x03) Body 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Message ID (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Message Length (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Unscheduled Sent (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Class | Weight | Service Hint | Binding Kind | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Deadline | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Binding Value (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Message ID is chosen by the sender as described in ; zero is reserved. Binding Kind and Binding Value are defined in . Class is 0..7; values above 7 MUST be treated as 7. Weight is 1..255; zero MUST be treated as 1. Service Hint values are registered in : 0 none, 1 latency-sensitive, 2 throughput-oriented. Deadline is the number of microseconds, relative to transmission of the request, by which the sender would like the message delivered; zero means none.

CREDIT_GRANT

CREDIT_GRANT (Type 0x04) Body 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Message ID (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Grant Limit (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Pacing Rate | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Lifetime | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Flags: 0x0001 POOL (Message ID is zero and the grant covers the association's pool), 0x0002 FINAL (Grant Limit equals the message length). Pacing Rate MUST NOT be zero; a sender that receives a grant with Pacing Rate zero MUST ignore it. Lifetime zero means the default Grant Lifetime of the association.

SENDER_STATUS

SENDER_STATUS (Type 0x05) Body 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Unused Credit (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Congestion Lvl | Service Hint | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Suggested Rate | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Suggested Rate zero means that the sender makes no suggestion. The Service Hint states the dominant hint of the sender's pending messages toward this receiver.

CONGESTION_REPORT

CONGESTION_REPORT (Type 0x06) Body 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reporter ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Queue Depth | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Queue Threshold | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Cong. Degree | Direction | AF | Protocol | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Port | Destination Port | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Suggested Rate | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Address (4 or 16 octets, per AF) ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Destination Address (4 or 16 octets, per AF) ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Reporter ID is a domain-unique identifier of the CRN. Queue Depth and Queue Threshold are in octets. Direction is 1 when the report is sent to the receiver and 2 when it is sent to the sender. AF is 4 for IPv4 (total length 48) or 6 for IPv6 (total length 72). Protocol, ports, and addresses are those of the data packet that triggered the report; for RoCEv2 the ports are the UDP ports of that packet.

MSG_DONE, CANCEL, CREDIT_RETURN, and ERROR MSG_DONE (0x07) carries Message ID (64 bits) and Octets Received (64 bits). CANCEL (0x08) carries Message ID (64 bits) and a Reason (16 bits) followed by 16 reserved bits. CREDIT_RETURN (0x09) carries Message ID (64 bits) and New Limit (64 bits), the offset beyond which the sender will not transmit; New Limit MUST NOT be smaller than the octets already transmitted. ERROR (0x0A) carries a Reason (16 bits), 16 reserved bits, and a Message ID (64 bits), which is zero when the error concerns the whole association. Reason codes are listed in ; CANCEL uses the same codes. RDCC Reason Codes

Code	Name
0x0000	NO_ERROR
0x0001	ASSOC_REFUSED
0x0002	UNKNOWN_ASSOCIATION
0x0003	BUDGET_EXHAUSTED
0x0004	UNSUPPORTED_TYPE
0x0005	MESSAGE_TOO_LARGE
0x0006	APPLICATION_ABORT
0x0007	UNSUPPORTED_BINDING

Sender Message State Machine Each message at the sender is in one of the states NEW, UNSCHED, WAITING, SCHEDULED, DRAINED, DONE, or ABORTED. defines the transitions. The receiver keeps the mirror state implicitly through its accounting. Sender Message State Transitions

State	Event	Action	Next State
NEW	length <= U and no unscheduled message outstanding	start random delay; send CREDIT_REQUEST with Unscheduled Sent = length	UNSCHED
NEW	otherwise	send CREDIT_REQUEST	WAITING
UNSCHED	all octets transmitted	none	DRAINED
UNSCHED	CONGESTION_REPORT or link pause	stop; request credit for the remainder	WAITING
WAITING	CREDIT_GRANT above transmitted offset	none	SCHEDULED
WAITING	request timeout	retransmit CREDIT_REQUEST	WAITING
SCHEDULED	Grant Limit reached before message end	none	WAITING
SCHEDULED	Lifetime elapsed	send CREDIT_REQUEST	WAITING
SCHEDULED	all octets transmitted	none	DRAINED
DRAINED	MSG_DONE	release state	DONE
DRAINED	request timeout	retransmit CREDIT_REQUEST	DRAINED
any but DONE or ABORTED	CANCEL for the message; ERROR for the message with reason MESSAGE_TOO_LARGE, APPLICATION_ABORT, or UNSUPPORTED_BINDING; or ERROR with reason ASSOC_REFUSED	release state	ABORTED
WAITING	ERROR BUDGET_EXHAUSTED	keep the request outstanding	WAITING
any but DONE or ABORTED	ERROR UNKNOWN_ASSOCIATION, or new Epoch in ASSOC_ACCEPT	drop credit; re-establish; send CREDIT_REQUEST	WAITING

In WAITING the sender holds no usable credit and retransmits CREDIT_REQUEST as described in . In DRAINED all octets have been transmitted and the sender waits for MSG_DONE; a receiver that receives CREDIT_REQUEST for a completed message answers with MSG_DONE again. If the data transport retransmits octets already transmitted, those octets remain within the existing Grant Limit, because the receiver has not counted them as received.

Congestion Control Considerations RDCC is a receiver-driven admission mechanism and is used together with the congestion control of the data transport, as required for UDP applications by . The rate at which a sender transmits is bounded by both. RDCC complies with the principles of as follows:

• The data a receiver admits is bounded by G at every instant, and the rate it admits is bounded by its access link capacity, so RDCC never causes more traffic than the receiver can absorb.
• Unscheduled data is bounded per association by U, limited to one message, randomized by J, and accounted within G_u. Unscheduled data stops on any congestion report.
• RDCC reacts to ECN, delay, and on-path reports by reducing rates multiplicatively.
• As a circuit breaker, a receiver that observes a loss rate of unscheduled data above a configured threshold (RECOMMENDED: one percent over one second) MUST set U to zero for all associations until the loss rate falls below one tenth of the threshold.

Because RDCC does not disable ECN or rely on PFC, the network retains its ordinary congestion feedback. Operators that previously disabled ECN marking for all-to-all traffic can re-enable it once receivers bound their incast through RDCC.

Operational Considerations Parameter guidance (all values are configurable): Default Parameters

Parameter	Guidance
G	1 to 2 times the access link BDP
G_u	at most G/4
U	G_u divided by expected concurrent senders; at most one BDP
J	1 to 2 times the base RTT
N_cu	2 times the sender's own BDP
Grant Lifetime	4 times the base RTT, at least 100 microseconds
B_th	the ECN marking threshold of the queue, if ECN is used
Credit Unit	path MTU payload size

Implementations SHOULD expose counters for: credit granted, reclaimed, and returned; unscheduled octets; reports received by type; and invariant violations. Deployment can be incremental: receivers that do not implement RDCC simply never answer ASSOC_REQUEST, and the sender then uses its transport without RDCC after its association attempts time out. CRNs are optional; without them, receivers rely on ECN and delay. RDCC control messages SHOULD be marked with a DSCP that receives at least the priority of the data they control, so that grants are not delayed behind the data they schedule.

Security Considerations The security considerations of apply. This section follows the guidance of . Off-path attackers. An attacker that can inject UDP datagrams could forge grants (to induce senders to overload a receiver), forge MSG_DONE or CANCEL (to disrupt messages), or forge reports (to reduce rates). RDCC limits the impact of blind injection with the random 64-bit Sender Nonce, the 32-bit Association ID, the Epoch, and sequence-number checks. These are not cryptographic protections. Deployments MUST therefore do at least one of the following: protect RDCC traffic with IPsec ESP between endpoints, or ensure, through ingress filtering at the boundary of the limited domain, that datagrams addressed to the RDCC port cannot enter the domain from outside. On-path and compromised nodes. A compromised CRN can only reduce rates or stop unscheduled transmission, because endpoints never increase credit or rate in response to a report (). The effect is bounded to a denial of service that the same node could cause by dropping packets. Misbehaving senders. A sender that transmits beyond its credit gains nothing: the receiver detects the excess, discards it, and stops granting (). A sender that hoards credit without sending reduces the receiver's available budget only until the Grant Lifetime expires. Receivers SHOULD cap the credit and the number of pending messages per association. Resource exhaustion. Receivers keep per-association and per-message state. Receivers MUST bound the number of associations and of pending messages, and SHOULD refuse new associations rather than evicting active ones. Reflection. ASSOC_REQUEST is padded to the size of ASSOC_ACCEPT, and receivers send all other messages only to the address bound when the association was accepted, so an attacker cannot direct grants to a third party. CRNs send reports to the addresses found in data packets they forward; within a limited domain with ingress filtering these addresses cannot be spoofed from outside, and the per-flow and per-port rate limits of bound the residual reflection risk from inside the domain. Information exposure. Grants and reports reveal traffic volumes between hosts. Within a limited domain this information is available to the operator through other means; where it is sensitive, ESP provides confidentiality.

IANA Considerations This document requests no port number; the RDCC port is configured per limited domain ().

RDCC Registries IANA is requested to create a new registry group named "Receiver-Driven Credit Control (RDCC) Parameters" containing the following registries. Each registration records the value, the name, and a reference. Unless stated otherwise, the policy is Specification Required for the lower three quarters of the value space, and the upper quarter is reserved for Private Use. The designated experts should check that a new value is fully specified, does not duplicate an existing one, and does not weaken the congestion control properties of .

• "RDCC Message Types" (8 bits), initial contents from ; 0x00 is Reserved.
• "RDCC Reason Codes" (16 bits), initial contents from .
• "RDCC Service Hints" (8 bits): 0 None, 1 Latency-Sensitive, 2 Throughput-Oriented.
• "RDCC Binding Kinds" (8 bits): 0 Reserved, 1 RoCEv2, 2 Opaque.
• "RDCC Capability Bits": one entry per bit of the 32-bit Capabilities field; initial contents 0x00000001 POOL, 0x00000002 STATUS, 0x00000004 CRN. All bits are assigned by Specification Required.
• "RDCC Message Flags": one entry per bit of the 16-bit Flags field and message type; initial contents ASSOC_ACCEPT 0x0001 Parameter Update, ASSOC_REQUEST 0x0002 Update Acknowledgment, CREDIT_GRANT 0x0001 POOL and 0x0002 FINAL. All bits are assigned by Specification Required.

References Normative References Informative References China Mobile China Mobile Centec Centec Centec Inesa SAIA China Telecom China Telecom China Telecom China Telecom MIT Lincoln Laboratory LabN Consulting, L.L.C. LabN Consulting, L.L.C. ViYouna China Academy of Information and Communications Technology Huawei Technologies China Telecom Google Google Meta IEEE Proceedings of ACM SIGCOMM 2015, pp. 523-536 Proceedings of ACM SIGCOMM 2017, pp. 29-42 Proceedings of ACM SIGCOMM 2018, pp. 221-235 InfiniBand Trade Association Ultra Ethernet Consortium

Example Exchange and Encodings A sender with a 4 MiB KV cache transfer and a 2 KiB dispatch message toward the same receiver, with U = 16 KiB, proceeds as follows: Sender Receiver | ASSOC_REQUEST(nonce) | |---------------------------------------------------->| | ASSOC_ACCEPT(assoc, U, J, N_cu, ...) | |<----------------------------------------------------| | (random delay in [0, J]) | | CREDIT_REQUEST(id=0x100, len=2048, unsched=2048) | | data 0..2047 (unscheduled DSCP) | |---------------------------------------------------->| | CREDIT_REQUEST(id=0x101, len=4 MiB, class 2) | |---------------------------------------------------->| | MSG_DONE(0x100) | | CREDIT_GRANT(0x101, limit=256 KiB, 100G) | |<----------------------------------------------------| | data 0..262143 of 0x101, paced | |---------------------------------------------------->| | CREDIT_GRANT(0x101, limit=512 KiB, 60G) | |<----------------------------------------------------| | ... | | CREDIT_GRANT(0x101, limit=4 MiB, FINAL) | |<----------------------------------------------------| | MSG_DONE(0x101) | |<----------------------------------------------------| The CREDIT_REQUEST for message 0x101 (Association ID 0x5A17C0DE, Epoch 3, Sequence Number 7, Class 2, Weight 4, Service Hint 1, RoCEv2 binding with destination queue pair 0x000123, first message sequence number 5, and 16 RDMA messages of 256 KiB each) is encoded as: 10 03 00 38 5a 17 c0 de 00 03 00 00 00 00 00 07 00 00 00 00 00 00 01 01 00 00 00 00 00 40 00 00 00 00 00 00 00 00 00 00 02 04 01 01 00 00 00 00 00 01 23 00 00 05 00 10 The first CREDIT_GRANT (Sequence Number 42, Grant Limit 262144, Pacing Rate 100000 Mbit/s, Lifetime 2000 microseconds) is encoded as: 10 04 00 28 5a 17 c0 de 00 03 00 00 00 00 00 2a 00 00 00 00 00 00 01 01 00 00 00 00 00 04 00 00 00 01 86 a0 00 00 07 d0

Acknowledgments The authors thank the participants of the distributed inference networking discussions for their review of the problem statement.