China Mobile

Beijing100053 China lizhiqiangyjy@chinamobile.com

China Mobile

Beijing100053 China duzongpeng@chinamobile.com

Centec

Shanghai201203 China wangjj@centec.com

Centec

Shanghai201203 China chengw@centec.com

Centec

Shanghai200000 China zhaoying@centec.com

SAIA

Shanghai200125 China chunhao.zhao@sh-aia.com

Inesa

Shanghai200030 China sunxun@inesa.com

Web and Internet Transport RDMA RoCEv2 forward erasure correction coded retransmission Alternate Marking long-distance high BDP RDMA transports designed for lossless data center fabrics perform poorly over long-distance, high bandwidth-delay product, or lossy IP paths: go-back-N retransmission resends whole windows, selective repeat costs at least a round trip per loss, and fixed forward erasure correction wastes capacity when the path is clean. This document specifies the Congestion-Aware Reliability Enhancement Mechanism (CREM), which protects datagram-based RDMA flows without modifying their packets. CREM adds a UDP-encapsulated repair channel on which the sender transmits erasure-coded repair symbols computed over blocks of RDMA packets, adapts the redundancy to measured loss and to whether losses are caused by congestion, and recovers losses that exceed the redundancy with a coded retransmission: a number of repair symbols, computed only over the lost packets, that is smaller than the number of lost packets. Per-packet retransmission remains as the final fallback. Optional path reports from on-path nodes, derived from Alternate Marking measurements or from the nodes' own drops, inform the adaptation.

Introduction

Background RDMA has become the dominant transport for training, inference, and storage traffic inside AI data centers. Deployments now extend RDMA across data centers and metropolitan or wide-area links, for example to move key-value caches between inference sites or to synchronize training across sites . These paths have round-trip times of milliseconds rather than microseconds, carry several megabytes of data per round trip, and are not lossless. Reliability mechanisms behave very differently on such paths:

• Go-back-N retransmission, the classic behavior of RDMA reliable connections, resends every packet after a loss. A single loss on a high-BDP path wastes up to a full window.
• Selective repeat, using SACK or NACK, resends only lost packets but adds at least one round trip of latency per loss event and requires per-packet state.
• Forward erasure correction (FEC) sends redundancy in advance and recovers losses without waiting, but a fixed redundancy level either wastes capacity on clean paths or fails on bad ones.

Which mechanism performs best depends on message size, loss rate, and round-trip time, and these change over time. No interoperable mechanism today combines the strengths of retransmission and redundancy for RDMA over IP and adapts between them.

Overview of CREM CREM has three cooperating parts:

1. Adaptive forward erasure correction (): the sender groups the packets of an RDMA flow into source blocks of k consecutive packet sequence number positions and sends m repair symbols per block on a separate UDP flow. The pair (k, m) is negotiated and changed over a reliable control exchange as the observed loss changes.
2. Coded retransmission (): when a block has lost more packets than its repair symbols can recover, the receiver requests repair, listing the lost packets. The sender answers with new repair symbols computed over the lost packets only; their number is the number of lost packets minus the repair symbols the receiver already holds. If that also fails, CREM falls back to per-packet retransmission.
3. Congestion-aware path feedback (): receivers report loss and ECN statistics, and on-path nodes optionally report per-hop loss and delay measured with Alternate Marking , as well as packets they dropped. The sender uses this to tell congestion loss from transmission loss: redundancy is increased for the latter but not for the former, where it would add to the congestion.

CREM does not change RDMA packets, RDMA transport semantics, or congestion control. Repair traffic is rate-limited and reacts to congestion as specified in .

Scope and Applicability This document specifies CREM for RDMA transports that carry each packet in a UDP datagram and number packets with a packet sequence number (PSN), with RDMA over Converged Ethernet version 2 (RoCEv2) as the defined flow kind. CREM only reads the PSN and the destination queue pair of protected packets. RDMA transports over TCP, such as RDMAP with MPA , are reliable at the TCP layer and do not need CREM. CREM is intended for limited domains , such as interconnected data centers of one operator. The CREM sender and receiver functions can be implemented in the RDMA NIC, in a host shim, or in a gateway at the boundary of a long-distance segment that protects all RDMA flows crossing it.

Relationship to Other Work The FEC Framework (FECFRAME) defines how FEC schemes protect arbitrary packet flows; CREM follows its model of source blocks and repair flows and uses its construction of source symbols from packets, but adds adaptive parameters, coded retransmission, and RDMA-specific source identification. applies FECFRAME to bundle transfer; and add FEC to QUIC. extends acknowledgments inside the RDMA transport header; CREM leaves transport headers unchanged and can coexist with it. and address fast network notifications for RDMA and AI traffic; the path reports of CREM are specific to repair decisions and could be carried by a general notification mechanism when one is standardized. discusses loss localization with Alternate Marking.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Protected flow:

The packets of one RDMA connection in one direction, for example all RoCEv2 packets toward one destination queue pair.

CREM Sender Function (CSF), CREM Receiver Function (CRF):

The functions at the two ends of a protected flow that implement this document.

Source packet:

A packet of the protected flow, unchanged by CREM.

Source block:

k source packets with consecutive PSNs, starting at the Block Start PSN.

Source symbol:

The octet string derived from a source packet for coding ().

Repair symbol:

A coded combination of source symbols, carried in a REPAIR message.

Repair Index:

The identifier i of a repair symbol within its block.

Coded retransmission:

Repair symbols sent in response to a repair request and computed over the lost source symbols only.

Repair Session ID:

An identifier that links a coded retransmission to the request that triggered it.

Path Reporter:

An on-path node that implements .

Architecture

CREM Functions and Message Paths +----------+ source packets (unchanged) +----------+ +----------+ | RDMA |--------------------------->| |->| RDMA | | sender | | Path | | receiver | +----------+ | Reporter | +-----^----+ | CSF |--- REPAIR (CREM port) ---->| (option) |->| CRF | +----------+ +----------+ +----------+ restored packets go to the RDMA receiver via the CRF CREM messages (UDP, CREM port): CSF -> CRF : HELLO, PARAM_UPDATE, REPAIR, CLOSE CRF -> CSF : HELLO_ACK, PARAM_ACK, LOSS_REPORT, REPAIR_REQUEST, CLOSE Reporter -> CSF : PATH_REPORT

The CSF observes the protected flow as the RDMA sender emits it, forms source blocks, and sends repair symbols. The CRF observes the protected flow as it arrives, holds the source symbols of incomplete blocks, decodes when possible, and passes restored packets to the RDMA receiver as if they had arrived from the network. All CREM messages use UDP with the CREM port as destination port. The CREM port is a UDP port configured by the operator of the limited domain; this document does not request a port assignment from the "Service Name and Transport Protocol Port Number Registry" . All CSFs, CRFs, and Path Reporters of a domain MUST be configured with the same CREM port, because Path Reporters send to CSFs without prior signaling. Operators SHOULD choose the port from the Dynamic Ports range (49152-65535) and MUST ensure that no other service of the domain uses it. Implementations MUST NOT assume a default port value.

Source Blocks and Coding

Source Symbols For a source packet whose UDP payload is the octet string P (for RoCEv2, from the Base Transport Header through the invariant CRC), the source symbol is: S = uint16(len(P)) || P || zero padding to T octets where T, the symbol length, is at least the largest UDP payload of the flow plus 2. This follows the construction of application data unit information in without the flow identifier octet, which CREM conveys in the message header. The length prefix lets the CRF restore P exactly.

Source Blocks A source block covers k consecutive PSN positions, starting at the Block Start PSN; position j (0 <= j < k) corresponds to PSN Block Start PSN + j, modulo 2^24. Blocks of a flow do not overlap. Not every PSN position carries a packet in the protected direction: for example, an RDMA READ request consumes one PSN for each expected response packet, and those PSNs never appear in the request direction. The CSF therefore records which positions of a block carry a source packet in a Present Bitmap, which it sends in every REPAIR message of the block. Absent positions are neither coded nor expected; they are treated as zero symbols that are known to both sides. Only packets that carry a new PSN are sources; acknowledgment and negative acknowledgment packets are not protected. A source packet that the RDMA transport retransmits with the same PSN occupies the same position. Retransmitted packets are expected to be identical to the original; restored packets are verified as described in , so a difference is detected rather than delivered. The CSF closes a block when k positions have been covered, or earlier at the end of a burst; the k of a closed block is carried in its REPAIR messages. When the CRF has received no REPAIR message for a block, it does not know its boundaries and reports losses by PSN range instead (). The CRF does not need REPAIR messages to recognize the most common absent positions: when it observes an RDMA READ request, it derives the number of PSNs the request consumes from its DMA Length and the RDMA MTU announced in HELLO, and treats those PSNs as absent. If the CSF receives a PSN-range request that names absent positions, it answers with an INFO message: a REPAIR with the INFO flag, Symbol Length zero, and no symbol, which carries the Block Start PSN, k, and Present Bitmap of the block.

FEC Schemes FEC schemes are identified by an 8-bit value (). Support for scheme 1 is REQUIRED.

Scheme 0: Source Copy The repair symbol with Repair Index i is source symbol i of the block, unmodified. It is used for per-packet retransmission by the CSF ().

Scheme 1: Cauchy Reed-Solomon over GF(2^8) Arithmetic is in GF(2^8) defined by the primitive polynomial x^8 + x^4 + x^3 + x^2 + 1 (0x11D); addition is bitwise exclusive-or. For source index j and Repair Index i, define y_j = j and x_i = 255 - i, and the coefficient C(i, j) = 1 / (x_i + y_j) (computed in GF(2^8)) which exists whenever i + j < 255. The CSF MUST use only Repair Indices with i + k <= 255. A repair symbol over a set L of source indices is, octet by octet: R_i[L] = sum over j in L of C(i, j) * S_j An ordinary repair symbol uses L equal to all present positions of the block. A coded retransmission uses L equal to the set of lost sources named in the request (). Because every square submatrix of a Cauchy matrix is non-singular, the receiver can solve for any set U of u unknown source symbols from any u repair symbols of distinct indices whose sets L all contain U, after subtracting the contributions of the source symbols it holds. Ordinary repair symbols always satisfy this condition. A coded retransmission satisfies it because its L is the CRF's set of unknown sources at the time of the request, and that set only shrinks afterwards. Repair symbols generated without a request from the CRF () MUST therefore be ordinary repair symbols. Under these rules the code is maximum distance separable: as many symbols as there are present sources recover the block.

Scheme 2: XOR Parity The single repair symbol (m = 1, Repair Index 0) is the bitwise exclusive-or of all source symbols of the block. It recovers one lost source. Coded retransmission is not defined for this scheme; blocks with more than one loss use the fallback of .

Adaptive Forward Erasure Correction

Capability Negotiation Before protecting a flow, the CSF sends HELLO to the CRF, giving the flow descriptor, the schemes it supports, and its limits for k, m, and T. The CRF answers with HELLO_ACK, which assigns the Flow ID used in all later messages for the flow and states the CRF's limits, including the number of blocks it can hold concurrently. The CRF MUST choose the Flow ID at random among unused values. The CSF retransmits HELLO with exponential backoff, starting at 1 ms, for at most eight attempts; without an answer it does not protect the flow. The effective limits are the minimum of the two sides. The CSF sends all later messages of the flow to the address and port from which HELLO_ACK came, and the CRF sends to the address and port from which HELLO came.

Parameter Synchronization Each configuration (scheme, k, m, T) has a Config ID. To change the configuration, the CSF sends PARAM_UPDATE with a new Config ID and an Effective PSN. It retransmits PARAM_UPDATE after a retransmission timeout (initially 2 * SRTT, at least 1 ms), doubling the timeout on each retransmission, until it receives PARAM_ACK for that Config ID; after eight unanswered attempts it sends CLOSE. The CSF MUST NOT use the new configuration for blocks that start before the Effective PSN or before PARAM_ACK has been received. If the Effective PSN passes before the acknowledgment arrives, the CSF continues with the old configuration and sends PARAM_UPDATE again with a later Effective PSN. Every REPAIR message carries its Config ID, scheme, and k, so that the CRF never decodes a block with the wrong parameters. The CRF reserves decoding resources when it acknowledges a configuration and MUST accept REPAIR messages for any configuration it has acknowledged and not yet released. A configuration is released when a newer configuration has been acknowledged and every block that uses the older one has been closed or has fallen back.

Redundancy Policy The redundancy policy is local to the CSF, subject to the following requirements:

• The CSF MUST classify recent losses as congestion-related or not, using the evidence of : rising ECN Congestion Experienced marks , growth of queueing delay reported for a hop, and drops reported by a node as queue overflow indicate congestion; losses without these signals are treated as transmission losses.
• The CSF MUST NOT increase m in response to congestion-related loss beyond a configured m_cong (RECOMMENDED: 1). Redundancy does not help a congested queue and adds load to it.
• The CSF MUST apply hysteresis: it MUST NOT change the configuration more than once per two smoothed round-trip times, because a change takes effect only after a round trip.

The RECOMMENDED policy for transmission loss uses an estimate p of the packet loss rate (an exponentially weighted moving average over LOSS_REPORT intervals) and chooses the smallest m for which the probability that a block of k + m packets loses more than m packets, under independent losses with rate p, is below a target epsilon (default 10^-4). m = 0 is chosen when p is so low that coded retransmission alone meets the target; the flow then incurs no redundancy overhead and still benefits from coded retransmission. The block size k trades decoding delay (the CRF waits for up to k packets) against overhead; k = 32 is a reasonable default for multi-millisecond paths.

Sending Repair Symbols The CSF sends the m repair symbols of a block, with Repair Indices 0 to m-1, as soon as the block is closed. It MAY spread them among the next source packets to reduce burstiness. Repair messages are subject to the rate limits of .

Receiver Operation For each open block, the CRF records the source symbols and repair symbols received. The following rules apply:

• When the number of distinct symbols received for a block reaches the number of present positions, the CRF decodes the missing source symbols and restores the packets. It MUST verify the invariant CRC of each restored RoCEv2 packet and discard a packet that fails; it then requests a fallback for that position (). Verified packets are passed to the RDMA receiver, and the block is closed.
• The CRF MAY hold source packets with PSNs after a gap for up to a hold time H (RECOMMENDED: the time to receive one block at the flow rate, at most one quarter of the minimum RTT), so that restored packets reach the RDMA receiver in order. When H expires, it MUST release held packets in PSN order even if the gap remains, so that CREM never delays the RDMA transport's own recovery by more than H.
• Packets restored after their successors were released are still passed to the RDMA receiver; RDMA transports discard duplicates and accept late packets according to their own rules.
• A block is incomplete when, after the CRF has observed a packet beyond the block or all of its repair symbols, plus a reordering allowance (RECOMMENDED: H), it holds fewer symbols than present positions. The CRF then proceeds as in . A PSN gap for which the CRF has seen no REPAIR message is handled in the same way, using the PSN-range form of the request.

Restored packets carry no ECN marking. The CRF counts them separately so that they do not mask losses in its LOSS_REPORT.

Coded Retransmission

Repair Request For an incomplete block, the CRF sends REPAIR_REQUEST with the Block Start PSN, k, the Config ID, a bitmap of the lost source indices (the set L, with X = |L| lost sources), the number r of repair symbols it holds for the block, and a suggested count M_s = X - r + delta, where delta (RECOMMENDED: 0 on clean paths, 1 when p exceeds one percent) allows for the loss of repair symbols; M_s MUST NOT exceed X. The final decision on M rests with the CSF. A Request ID distinguishes successive requests for the same block. When the CRF has received no REPAIR message for the affected positions, it does not know the block structure and sends the PSN-range form instead: a list of lost PSN ranges, each a start PSN and a count. This list of lost PSNs is all the CSF needs: it maps the PSNs to its blocks and answers each block as below, with r taken as zero.

Coded Response On receiving REPAIR_REQUEST for a block it still holds, the CSF:

1. chooses M with X - r <= M <= X, normally M = M_s;
2. computes M repair symbols R_i[L] over the lost set L, using Repair Indices not previously used for the block; if fewer than M unused indices with i + k <= 255 remain, it answers with Source Copy symbols instead ();
3. sends them in REPAIR messages with the RETX flag set, the lost-set bitmap, and a Repair Session ID equal to the Request ID; and
4. sends them back to back, so that the whole repair takes one round trip regardless of X.

Because M is X - r (plus delta), coded retransmission sends fewer symbols than the number of lost packets whenever the receiver holds at least one repair symbol of the block, and it recovers all lost packets within one round trip. The CSF MUST keep the source symbols of a block until the block is acknowledged by the RDMA transport or until it gives up retransmission for the block. It MUST respond to at most one request per block per smoothed round-trip time. The CRF decodes when the ordinary and retransmitted repair symbols it holds, together with its source symbols, number at least the present positions of the block. If source packets in L arrive late, the CRF subtracts their contribution from the retransmitted symbols as well.

Fallback A block that remains incomplete after two coded retransmission rounds, or whose scheme does not support coded retransmission, falls back to per-packet retransmission. Either:

• the CSF sends the lost source symbols with scheme 0 (Source Copy) in response to a REPAIR_REQUEST carrying the FALLBACK flag; or
• the CRF releases all held packets of the block and leaves recovery to the RDMA transport's own retransmission.

In both cases the data is eventually delivered by the RDMA transport's reliability, which CREM never disables.

Congestion-Aware Path Feedback

Receiver Loss Reports The CRF sends LOSS_REPORT at least once per smoothed round-trip time while the flow is active. It reports, for the interval: source packets expected and received (before restoration), packets restored by decoding, blocks incomplete after the first decoding attempt, ECN CE marks observed on source packets, REPAIR messages received, CE marks observed on REPAIR messages, and the interval duration. The CSF derives the path loss rate p and the CE fractions from it.

On-Path Path Reports Support for this section is OPTIONAL for network nodes; CSFs MUST be able to process PATH_REPORT. A Path Reporter that measures the protected flow with Alternate Marking (, , applied to IPv6 with ) can determine, per marking period, how many packets of the flow were lost between measurement points and how its queueing delay changed. A node also knows exactly which packets it dropped itself, for example because of queue overflow or a corrupted frame. A Path Reporter MAY send PATH_REPORT to the source address of the protected flow, containing one Hop Entry per measurement point with:

• the node identifier and the marking period number;
• the number of packets lost in the period;
• the change in queueing delay in microseconds, signed;
• a cause code: 1 for queue overflow, 2 for transmission error, 0 for unknown; and
• optionally, the PSNs of packets the node dropped itself.

A CSF that receives PSNs dropped by a node MAY send additional ordinary repair symbols for the affected blocks without waiting for the CRF's request (). This saves the time between the drop and the receiver's detection of it. Path Reporters send PATH_REPORT to the CREM port of the CSF address configured for the flow's source prefix, or, by default, to the flow's source address. A CSF in a gateway on the path toward the source intercepts reports addressed to the sources it serves. A PATH_REPORT MUST fit in the path MTU; a reporter truncates the dropped-PSN lists and sets the Truncated flag (0x80) if needed. When reverse-path capacity is scarce, a Path Reporter that forwards PATH_REPORT messages from downstream reporters for the same CSF MAY merge their Hop Entries with its own into one PATH_REPORT, up to 16 entries, within an aggregation interval (RECOMMENDED: 50 microseconds). Reporters MUST rate-limit PATH_REPORT per flow and per port. Alternate Marking, where it is deployed, is a measurement function of the network operator and is applied independently of CREM, for example in IPv6 options added and removed at the edges of the measured domain . CREM itself does not mark or modify protected packets. A CSF MUST accept PATH_REPORT only from the set of reporter addresses configured for the domain. Path reports influence only redundancy, the timing of retransmission, and loss classification; they never cause the CSF to deem data delivered.

Message Formats

Common Header

CREM Common Header 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Version| Type | Flags | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Flow ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Version is 1. Length is the length of the message in octets, including this header. Flow ID is assigned by the CRF in HELLO_ACK and is zero in HELLO and in PATH_REPORT. Integers are in network byte order. Types: 1 REPAIR, 2 HELLO, 3 HELLO_ACK, 4 PARAM_UPDATE, 5 PARAM_ACK, 6 REPAIR_REQUEST, 7 LOSS_REPORT, 8 PATH_REPORT, 9 CLOSE. Messages with an unknown Version or Type MUST be discarded.

REPAIR

REPAIR Message (Type 1) after the Common Header 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Config ID | Scheme | k | Repair Index | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Block Start PSN | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Repair Session ID | Symbol Length T | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Present Bitmap (ceil(k/32) 32-bit words) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Lost Source Bitmap (ceil(k/32) words, RETX flag only) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Repair Symbol (T octets) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Flags: 0x80 RETX (coded retransmission; the Lost Source Bitmap is present), 0x40 LAST (last repair symbol the CSF intends to send for this block in this round), 0x20 INFO (block information only; Symbol Length is zero and no symbol follows). Repair Session ID is zero for ordinary repair symbols. In both bitmaps, bit j (counting from the most significant bit of the first word) refers to position j; in the Present Bitmap it is set if position j carries a source packet, and in the Lost Source Bitmap if source j is in L. k is 1 to 255.

HELLO and HELLO_ACK

HELLO (Type 2) after the Common Header 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Flow Kind | AF | Supported Schemes | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Max k | Max m | Max T | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | RDMA MTU | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sender Nonce (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Address (4 or 16 octets, per AF) ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Destination Address (4 or 16 octets, per AF) ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Flow Label | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Flow Kind is 1 for RoCEv2. AF is 4 or 6. Supported Schemes is a bitmap with bit s (from the least significant bit) set if scheme s is supported. RDMA MTU is the path MTU of the RDMA connection in octets of payload per packet. For RoCEv2 the Flow Label holds the destination queue pair number in its low 24 bits. HELLO_ACK (Type 3) carries the same first twelve octets with the CRF's Supported Schemes, Max k, Max m, and Max T (RDMA MTU echoed), followed by the echoed Sender Nonce, Max Open Blocks (16 bits), and 16 reserved bits; the Flow ID in its header is the assigned identifier. HELLO_ACK is therefore never larger than HELLO.

PARAM_UPDATE and PARAM_ACK PARAM_UPDATE (Type 4) carries Config ID (8 bits), Scheme (8 bits), k (8 bits), m (8 bits), T (16 bits), Reserved (16 bits), Effective PSN (24 bits), and Reserved (8 bits). PARAM_ACK (Type 5) carries Config ID (8 bits), Status (8 bits; 0 accepted, 1 unsupported scheme, 2 limits exceeded, 3 resources exhausted), and Reserved (16 bits). A CSF that receives a non-zero Status keeps its current configuration.

REPAIR_REQUEST REPAIR_REQUEST (Type 6) has two forms. The block form carries Config ID (8 bits), Scheme (8 bits), k (8 bits), Repairs Held r (8 bits), Block Start PSN (24 bits), Suggested M (8 bits), Request ID (16 bits), Reserved (16 bits), and the Lost Source Bitmap (ceil(k/32) words). The PSN-range form, indicated by flag 0x10 RANGES, carries Request ID (16 bits), Range Count (16 bits), and Range Count entries of Start PSN (24 bits) and Count (8 bits). Flag 0x20 FALLBACK requests Source Copy symbols for the lost sources.

LOSS_REPORT LOSS_REPORT (Type 7) carries Interval (32 bits, microseconds), Expected (32 bits), Received (32 bits), Restored (32 bits), Incomplete Blocks (32 bits), CE Marked (32 bits), Repairs Received (32 bits), and Repairs CE Marked (32 bits), all counts for the interval.

PATH_REPORT

PATH_REPORT (Type 8) after the Common Header 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Method | Entry Count | AF | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Flow source and destination address (4 or 16 octets each) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Flow Label | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Hop Entry 1 (see below) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Hop Entry: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Node ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Marking Period Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Lost Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Queue Delay Change (signed, us) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Cause | PSN Count | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved | Dropped PSN (24 bits), repeated PSN Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Method: 1 Alternate Marking , 2 Clustered Alternate Marking , 3 node-local drop accounting. The flow is identified by addresses and Flow Label as in HELLO, because a Path Reporter does not know the Flow ID. Entry Count is at most 16.

CLOSE CLOSE (Type 9) carries a Reason (16 bits) and Reserved (16 bits). Either side sends it to release the flow's state. Reasons: 0 normal termination, 1 flow ended, 2 parameter synchronization failed, 3 resources exhausted, 4 circuit breaker.

Congestion Control Considerations Repair traffic adds load to the network. The congestion control of an RDMA flow normally runs in the RDMA interface, which a CSF in a host shim or gateway cannot influence directly, and REPAIR messages use a different UDP port and can therefore take a different path under equal-cost multipath routing. CREM therefore limits repair traffic itself:

• The rate of ordinary repair symbols MUST NOT exceed m/k times the measured rate of the protected flow, and the volume of coded retransmissions MUST NOT exceed the volume of source packets reported lost.
• REPAIR messages carry the DSCP of the protected flow and are ECN capable. When LOSS_REPORT shows CE marks or loss on REPAIR messages, the CSF MUST reduce m (to zero if the condition persists for more than one second) in addition to applying the classification of .
• A CSF integrated with the RDMA interface SHOULD also count REPAIR octets against the flow's congestion control, so that redundancy trades the flow's own goodput for lower latency.

Because redundancy is not raised for congestion-related loss, CREM does not amplify congestion. These rules follow and . As a circuit breaker in the sense of , a CSF that observes a path loss rate (from LOSS_REPORT, before restoration) above ten percent or a CE fraction above fifty percent over one second MUST set m to zero and stop coded retransmission for the flow, leaving recovery to the RDMA transport. The flow stays open, so LOSS_REPORT messages continue, and the CSF resumes redundancy only after the condition has been absent for at least ten seconds. If the condition persists for more than one minute, the CSF sends CLOSE with reason 4.

Operational Considerations

• MTU: a REPAIR message is larger than a source packet by its header (20 octets plus both bitmaps) and the CREM UDP/IP headers. The symbol length T plus these headers MUST fit the path MTU.
• Memory: the CSF keeps the source symbols of unacknowledged blocks; the CRF keeps up to Max Open Blocks blocks. Both are bounded by the bandwidth-delay product of the path.
• Gateways: a pair of CREM gateways at the ends of a long-distance segment can protect all RDMA flows crossing it, so that end hosts need no changes.
• Monitoring: implementations SHOULD expose the configured (k, m), restored packets, coded retransmissions and their sizes, fallbacks, and the loss classification.

Security Considerations Data integrity is the main concern. A forged REPAIR message would make the CRF reconstruct wrong packets, which the RDMA receiver would place into application memory. The invariant CRC of RoCEv2 detects accidental corruption of restored packets but not a deliberate forgery. Deployments MUST therefore either protect CREM messages, and the protected flow, with IPsec ESP , or confine CREM to a limited domain whose boundary drops datagrams addressed to the CREM port from outside. The random Sender Nonce and Flow ID make blind injection unlikely but do not authenticate messages. Forged REPAIR_REQUEST or PATH_REPORT messages could cause the CSF to send additional repair traffic. The amount is bounded: at most X symbols per block per round trip, within the rate limits of ; REPAIR messages are sent only to the address bound at HELLO, never to the source of a request; and PATH_REPORT is accepted only from configured reporters. HELLO is no smaller than HELLO_ACK, so CREM cannot be used for reflection amplification. Path reports reveal per-hop loss and delay. They are sent only to sources of protected flows inside the domain.

IANA Considerations This document requests no port number; the CREM port is configured per limited domain ().

CREM Registries IANA is requested to create a registry group named "CREM Parameters" with the following registries. Unless stated otherwise, values are assigned by Specification Required and the upper quarter of each value space is reserved for Private Use.

• "CREM Message Types" (4 bits): 0 Reserved, 1 REPAIR, 2 HELLO, 3 HELLO_ACK, 4 PARAM_UPDATE, 5 PARAM_ACK, 6 REPAIR_REQUEST, 7 LOSS_REPORT, 8 PATH_REPORT, 9 CLOSE. Assignments require Standards Action.
• "CREM FEC Schemes" (8 bits): 0 Source Copy, 1 Cauchy Reed-Solomon GF(2^8), 2 XOR Parity.
• "CREM Flow Kinds" (8 bits): 0 Reserved, 1 RoCEv2.
• "CREM Measurement Methods" (8 bits): 0 Reserved, 1 Alternate Marking, 2 Clustered Alternate Marking, 3 Node-local drop accounting.
• "CREM Loss Causes" (8 bits): 0 Unknown, 1 Queue overflow, 2 Transmission error.
• "CREM PARAM_ACK Status Codes" (8 bits): 0 Accepted, 1 Unsupported scheme, 2 Limits exceeded, 3 Resources exhausted.
• "CREM CLOSE Reasons" (16 bits): 0 Normal termination, 1 Flow ended, 2 Parameter synchronization failed, 3 Resources exhausted, 4 Circuit breaker.
• "CREM Message Flags": one entry per bit of the 8-bit Flags field and message type; initial contents REPAIR 0x80 RETX, 0x40 LAST, and 0x20 INFO, REPAIR_REQUEST 0x20 FALLBACK and 0x10 RANGES, PATH_REPORT 0x80 Truncated.

The designated experts should verify that new FEC schemes specify the coding of absent positions and whether coded retransmission is supported, and that new flow kinds specify how sources and PSN positions are identified.

References Normative References Informative References China Mobile China Mobile Centec Centec Centec Inesa SAIA Aalyria Technologies UCLouvain UCLouvain, WEL RI Huawei Huawei Huawei Huawei Huawei China Mobile China Mobile China Telecom China Telecom China Telecom China Telecom China Academy of Information and Communications Technology Huawei Technologies China Telecom China Mobile China Mobile InfiniBand Trade Association

Coding Example This example uses scheme 1 with k = 4 and T = 16. The UDP payloads of the four source packets are the ASCII strings "RDMA-PKT-0" to "RDMA-PKT-3" (10 octets each), so each source symbol is the length 0x000a, the payload, and four octets of padding: S0 = 000a52444d412d504b542d3000000000 S1 = 000a52444d412d504b542d3100000000 S2 = 000a52444d412d504b542d3200000000 S3 = 000a52444d412d504b542d3300000000 Coefficients C(i, j) for i = 0..2 and j = 0..3 (x_i = 255 - i, y_j = j): j=0 j=1 j=2 j=3 i=0: fd 7e ff 7f i=1: 7e fd 7f ff i=2: ff 7f fd 7e With m = 2 the CSF sends R0 and R1 over the whole block: R0 = 001ef6ccd7c377f0ddfc774c00000000 R1 = 001ef6ccd7c377f0ddfc774f00000000 Sources 0, 1, and 2 are lost (X = 3); the CRF holds S3, R0, and R1 (r = 2) and requests M = X - r = 1 symbol for L = {0, 1, 2}. The CSF sends the coded retransmission with Repair Index 2 over L only: R2[L] = 00354fdf6d4b40b57e5c400000000000 The CRF subtracts C(0,3)*S3 from R0 and C(1,3)*S3 from R1 and solves the 3 x 3 Cauchy system with rows i = 0, 1, 2 and columns j = 0, 1, 2, recovering S0, S1, and S2. Three lost packets were recovered with one retransmitted symbol. The REPAIR message for R2[L] (Flow ID 0x0000BEEF, Config ID 1, Block Start PSN 0x001000, Repair Session ID 7, all four positions present, so Present Bitmap 0xF0000000 and Lost Source Bitmap 0xE0000000) begins as follows and is followed by the 16 octets of R2[L], for a total Length of 44: 11 80 00 2c 00 00 be ef 01 01 04 02 00 10 00 00 00 07 00 10 f0 00 00 00 e0 00 00 00

Acknowledgments The authors thank the participants in discussions on RDMA over long-distance networks for their comments.