| Internet-Draft | CREM for RDMA | November 2026 |
| Li, et al. | Expires 6 May 2027 | [Page] |
RDMA transports designed for lossless data center fabrics perform poorly over long-distance, high bandwidth-delay product, or lossy IP paths: go-back-N retransmission resends whole windows, selective repeat costs at least a round trip per loss, and fixed forward erasure correction wastes capacity when the path is clean. This document specifies the Congestion-Aware Reliability Enhancement Mechanism (CREM), which protects datagram-based RDMA flows without modifying their packets. CREM adds a UDP-encapsulated repair channel on which the sender transmits erasure-coded repair symbols computed over blocks of RDMA packets, adapts the redundancy to measured loss and to whether losses are caused by congestion, and recovers losses that exceed the redundancy with a coded retransmission: a number of repair symbols, computed only over the lost packets, that is smaller than the number of lost packets. Per-packet retransmission remains as the final fallback. Optional path reports from on-path nodes, derived from Alternate Marking measurements or from the nodes' own drops, inform the adaptation.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 6 May 2027.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
RDMA has become the dominant transport for training, inference, and storage traffic inside AI data centers. Deployments now extend RDMA across data centers and metropolitan or wide-area links, for example to move key-value caches between inference sites or to synchronize training across sites [I-D.li-tsvwg-inference-transport]. These paths have round-trip times of milliseconds rather than microseconds, carry several megabytes of data per round trip, and are not lossless.¶
Reliability mechanisms behave very differently on such paths:¶
Which mechanism performs best depends on message size, loss rate, and round-trip time, and these change over time. No interoperable mechanism today combines the strengths of retransmission and redundancy for RDMA over IP and adapts between them.¶
CREM has three cooperating parts:¶
CREM does not change RDMA packets, RDMA transport semantics, or congestion control. Repair traffic is rate-limited and reacts to congestion as specified in Section 10.¶
This document specifies CREM for RDMA transports that carry each packet in a UDP datagram and number packets with a packet sequence number (PSN), with RDMA over Converged Ethernet version 2 (RoCEv2) [IBTA-ROCEV2] as the defined flow kind. CREM only reads the PSN and the destination queue pair of protected packets. RDMA transports over TCP, such as RDMAP [RFC5040] with MPA [RFC5044], are reliable at the TCP layer and do not need CREM.¶
CREM is intended for limited domains [RFC8799], such as interconnected data centers of one operator. The CREM sender and receiver functions can be implemented in the RDMA NIC, in a host shim, or in a gateway at the boundary of a long-distance segment that protects all RDMA flows crossing it.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
+----------+ source packets (unchanged) +----------+ +----------+
| RDMA |--------------------------->| |->| RDMA |
| sender | | Path | | receiver |
+----------+ | Reporter | +-----^----+
| CSF |--- REPAIR (CREM port) ---->| (option) |->| CRF |
+----------+ +----------+ +----------+
restored packets go to the
RDMA receiver via the CRF
CREM messages (UDP, CREM port):
CSF -> CRF : HELLO, PARAM_UPDATE, REPAIR, CLOSE
CRF -> CSF : HELLO_ACK, PARAM_ACK, LOSS_REPORT,
REPAIR_REQUEST, CLOSE
Reporter -> CSF : PATH_REPORT
The CSF observes the protected flow as the RDMA sender emits it, forms source blocks, and sends repair symbols. The CRF observes the protected flow as it arrives, holds the source symbols of incomplete blocks, decodes when possible, and passes restored packets to the RDMA receiver as if they had arrived from the network. All CREM messages use UDP [RFC768] with the CREM port as destination port.¶
The CREM port is a UDP port configured by the operator of the limited domain; this document does not request a port assignment from the "Service Name and Transport Protocol Port Number Registry" [RFC6335]. All CSFs, CRFs, and Path Reporters of a domain MUST be configured with the same CREM port, because Path Reporters send to CSFs without prior signaling. Operators SHOULD choose the port from the Dynamic Ports range (49152-65535) and MUST ensure that no other service of the domain uses it. Implementations MUST NOT assume a default port value.¶
For a source packet whose UDP payload is the octet string P (for RoCEv2, from the Base Transport Header through the invariant CRC), the source symbol is:¶
S = uint16(len(P)) || P || zero padding to T octets¶
where T, the symbol length, is at least the largest UDP payload of the flow plus 2. This follows the construction of application data unit information in [RFC6363] without the flow identifier octet, which CREM conveys in the message header. The length prefix lets the CRF restore P exactly.¶
A source block covers k consecutive PSN positions, starting at the Block Start PSN; position j (0 <= j < k) corresponds to PSN Block Start PSN + j, modulo 2^24. Blocks of a flow do not overlap. Not every PSN position carries a packet in the protected direction: for example, an RDMA READ request consumes one PSN for each expected response packet, and those PSNs never appear in the request direction. The CSF therefore records which positions of a block carry a source packet in a Present Bitmap, which it sends in every REPAIR message of the block. Absent positions are neither coded nor expected; they are treated as zero symbols that are known to both sides.¶
Only packets that carry a new PSN are sources; acknowledgment and negative acknowledgment packets are not protected. A source packet that the RDMA transport retransmits with the same PSN occupies the same position. Retransmitted packets are expected to be identical to the original; restored packets are verified as described in Section 6, so a difference is detected rather than delivered. The CSF closes a block when k positions have been covered, or earlier at the end of a burst; the k of a closed block is carried in its REPAIR messages. When the CRF has received no REPAIR message for a block, it does not know its boundaries and reports losses by PSN range instead (Section 7.1).¶
The CRF does not need REPAIR messages to recognize the most common absent positions: when it observes an RDMA READ request, it derives the number of PSNs the request consumes from its DMA Length and the RDMA MTU announced in HELLO, and treats those PSNs as absent. If the CSF receives a PSN-range request that names absent positions, it answers with an INFO message: a REPAIR with the INFO flag, Symbol Length zero, and no symbol, which carries the Block Start PSN, k, and Present Bitmap of the block.¶
FEC schemes are identified by an 8-bit value (Section 13). Support for scheme 1 is REQUIRED.¶
The repair symbol with Repair Index i is source symbol i of the block, unmodified. It is used for per-packet retransmission by the CSF (Section 7.3).¶
Arithmetic is in GF(2^8) defined by the primitive polynomial x^8 + x^4 + x^3 + x^2 + 1 (0x11D); addition is bitwise exclusive-or. For source index j and Repair Index i, define y_j = j and x_i = 255 - i, and the coefficient¶
C(i, j) = 1 / (x_i + y_j) (computed in GF(2^8))¶
which exists whenever i + j < 255. The CSF MUST use only Repair Indices with i + k <= 255. A repair symbol over a set L of source indices is, octet by octet:¶
R_i[L] = sum over j in L of C(i, j) * S_j¶
An ordinary repair symbol uses L equal to all present positions of the block. A coded retransmission uses L equal to the set of lost sources named in the request (Section 7). Because every square submatrix of a Cauchy matrix is non-singular, the receiver can solve for any set U of u unknown source symbols from any u repair symbols of distinct indices whose sets L all contain U, after subtracting the contributions of the source symbols it holds. Ordinary repair symbols always satisfy this condition. A coded retransmission satisfies it because its L is the CRF's set of unknown sources at the time of the request, and that set only shrinks afterwards. Repair symbols generated without a request from the CRF (Section 8.2) MUST therefore be ordinary repair symbols. Under these rules the code is maximum distance separable: as many symbols as there are present sources recover the block.¶
The single repair symbol (m = 1, Repair Index 0) is the bitwise exclusive-or of all source symbols of the block. It recovers one lost source. Coded retransmission is not defined for this scheme; blocks with more than one loss use the fallback of Section 7.3.¶
Before protecting a flow, the CSF sends HELLO to the CRF, giving the flow descriptor, the schemes it supports, and its limits for k, m, and T. The CRF answers with HELLO_ACK, which assigns the Flow ID used in all later messages for the flow and states the CRF's limits, including the number of blocks it can hold concurrently. The CRF MUST choose the Flow ID at random among unused values. The CSF retransmits HELLO with exponential backoff, starting at 1 ms, for at most eight attempts; without an answer it does not protect the flow. The effective limits are the minimum of the two sides. The CSF sends all later messages of the flow to the address and port from which HELLO_ACK came, and the CRF sends to the address and port from which HELLO came.¶
Each configuration (scheme, k, m, T) has a Config ID. To change the configuration, the CSF sends PARAM_UPDATE with a new Config ID and an Effective PSN. It retransmits PARAM_UPDATE after a retransmission timeout (initially 2 * SRTT, at least 1 ms), doubling the timeout on each retransmission, until it receives PARAM_ACK for that Config ID; after eight unanswered attempts it sends CLOSE. The CSF MUST NOT use the new configuration for blocks that start before the Effective PSN or before PARAM_ACK has been received. If the Effective PSN passes before the acknowledgment arrives, the CSF continues with the old configuration and sends PARAM_UPDATE again with a later Effective PSN. Every REPAIR message carries its Config ID, scheme, and k, so that the CRF never decodes a block with the wrong parameters.¶
The CRF reserves decoding resources when it acknowledges a configuration and MUST accept REPAIR messages for any configuration it has acknowledged and not yet released. A configuration is released when a newer configuration has been acknowledged and every block that uses the older one has been closed or has fallen back.¶
The redundancy policy is local to the CSF, subject to the following requirements:¶
The RECOMMENDED policy for transmission loss uses an estimate p of the packet loss rate (an exponentially weighted moving average over LOSS_REPORT intervals) and chooses the smallest m for which the probability that a block of k + m packets loses more than m packets, under independent losses with rate p, is below a target epsilon (default 10^-4). m = 0 is chosen when p is so low that coded retransmission alone meets the target; the flow then incurs no redundancy overhead and still benefits from coded retransmission. The block size k trades decoding delay (the CRF waits for up to k packets) against overhead; k = 32 is a reasonable default for multi-millisecond paths.¶
The CSF sends the m repair symbols of a block, with Repair Indices 0 to m-1, as soon as the block is closed. It MAY spread them among the next source packets to reduce burstiness. Repair messages are subject to the rate limits of Section 10.¶
For each open block, the CRF records the source symbols and repair symbols received. The following rules apply:¶
Restored packets carry no ECN marking. The CRF counts them separately so that they do not mask losses in its LOSS_REPORT.¶
For an incomplete block, the CRF sends REPAIR_REQUEST with the Block Start PSN, k, the Config ID, a bitmap of the lost source indices (the set L, with X = |L| lost sources), the number r of repair symbols it holds for the block, and a suggested count M_s = X - r + delta, where delta (RECOMMENDED: 0 on clean paths, 1 when p exceeds one percent) allows for the loss of repair symbols; M_s MUST NOT exceed X. The final decision on M rests with the CSF. A Request ID distinguishes successive requests for the same block.¶
When the CRF has received no REPAIR message for the affected positions, it does not know the block structure and sends the PSN-range form instead: a list of lost PSN ranges, each a start PSN and a count. This list of lost PSNs is all the CSF needs: it maps the PSNs to its blocks and answers each block as below, with r taken as zero.¶
On receiving REPAIR_REQUEST for a block it still holds, the CSF:¶
Because M is X - r (plus delta), coded retransmission sends fewer symbols than the number of lost packets whenever the receiver holds at least one repair symbol of the block, and it recovers all lost packets within one round trip. The CSF MUST keep the source symbols of a block until the block is acknowledged by the RDMA transport or until it gives up retransmission for the block. It MUST respond to at most one request per block per smoothed round-trip time.¶
The CRF decodes when the ordinary and retransmitted repair symbols it holds, together with its source symbols, number at least the present positions of the block. If source packets in L arrive late, the CRF subtracts their contribution from the retransmitted symbols as well.¶
A block that remains incomplete after two coded retransmission rounds, or whose scheme does not support coded retransmission, falls back to per-packet retransmission. Either:¶
In both cases the data is eventually delivered by the RDMA transport's reliability, which CREM never disables.¶
The CRF sends LOSS_REPORT at least once per smoothed round-trip time while the flow is active. It reports, for the interval: source packets expected and received (before restoration), packets restored by decoding, blocks incomplete after the first decoding attempt, ECN CE marks observed on source packets, REPAIR messages received, CE marks observed on REPAIR messages, and the interval duration. The CSF derives the path loss rate p and the CE fractions from it.¶
Support for this section is OPTIONAL for network nodes; CSFs MUST be able to process PATH_REPORT.¶
A Path Reporter that measures the protected flow with Alternate Marking ([RFC9341], [RFC9342], applied to IPv6 with [RFC9343]) can determine, per marking period, how many packets of the flow were lost between measurement points and how its queueing delay changed. A node also knows exactly which packets it dropped itself, for example because of queue overflow or a corrupted frame. A Path Reporter MAY send PATH_REPORT to the source address of the protected flow, containing one Hop Entry per measurement point with:¶
A CSF that receives PSNs dropped by a node MAY send additional ordinary repair symbols for the affected blocks without waiting for the CRF's request (Section 4.3.2). This saves the time between the drop and the receiver's detection of it.¶
Path Reporters send PATH_REPORT to the CREM port of the CSF address configured for the flow's source prefix, or, by default, to the flow's source address. A CSF in a gateway on the path toward the source intercepts reports addressed to the sources it serves. A PATH_REPORT MUST fit in the path MTU; a reporter truncates the dropped-PSN lists and sets the Truncated flag (0x80) if needed. When reverse-path capacity is scarce, a Path Reporter that forwards PATH_REPORT messages from downstream reporters for the same CSF MAY merge their Hop Entries with its own into one PATH_REPORT, up to 16 entries, within an aggregation interval (RECOMMENDED: 50 microseconds). Reporters MUST rate-limit PATH_REPORT per flow and per port.¶
Alternate Marking, where it is deployed, is a measurement function of the network operator and is applied independently of CREM, for example in IPv6 options added and removed at the edges of the measured domain [RFC9343]. CREM itself does not mark or modify protected packets.¶
A CSF MUST accept PATH_REPORT only from the set of reporter addresses configured for the domain. Path reports influence only redundancy, the timing of retransmission, and loss classification; they never cause the CSF to deem data delivered.¶
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Version| Type | Flags | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Flow ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Version is 1. Length is the length of the message in octets, including this header. Flow ID is assigned by the CRF in HELLO_ACK and is zero in HELLO and in PATH_REPORT. Integers are in network byte order. Types: 1 REPAIR, 2 HELLO, 3 HELLO_ACK, 4 PARAM_UPDATE, 5 PARAM_ACK, 6 REPAIR_REQUEST, 7 LOSS_REPORT, 8 PATH_REPORT, 9 CLOSE. Messages with an unknown Version or Type MUST be discarded.¶
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Config ID | Scheme | k | Repair Index | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Block Start PSN | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Repair Session ID | Symbol Length T | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Present Bitmap (ceil(k/32) 32-bit words) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Lost Source Bitmap (ceil(k/32) words, RETX flag only) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Repair Symbol (T octets) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Flags: 0x80 RETX (coded retransmission; the Lost Source Bitmap is present), 0x40 LAST (last repair symbol the CSF intends to send for this block in this round), 0x20 INFO (block information only; Symbol Length is zero and no symbol follows). Repair Session ID is zero for ordinary repair symbols. In both bitmaps, bit j (counting from the most significant bit of the first word) refers to position j; in the Present Bitmap it is set if position j carries a source packet, and in the Lost Source Bitmap if source j is in L. k is 1 to 255.¶
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Flow Kind | AF | Supported Schemes | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Max k | Max m | Max T | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | RDMA MTU | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sender Nonce (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Address (4 or 16 octets, per AF) ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Destination Address (4 or 16 octets, per AF) ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Flow Label | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Flow Kind is 1 for RoCEv2. AF is 4 or 6. Supported Schemes is a bitmap with bit s (from the least significant bit) set if scheme s is supported. RDMA MTU is the path MTU of the RDMA connection in octets of payload per packet. For RoCEv2 the Flow Label holds the destination queue pair number in its low 24 bits. HELLO_ACK (Type 3) carries the same first twelve octets with the CRF's Supported Schemes, Max k, Max m, and Max T (RDMA MTU echoed), followed by the echoed Sender Nonce, Max Open Blocks (16 bits), and 16 reserved bits; the Flow ID in its header is the assigned identifier. HELLO_ACK is therefore never larger than HELLO.¶
PARAM_UPDATE (Type 4) carries Config ID (8 bits), Scheme (8 bits), k (8 bits), m (8 bits), T (16 bits), Reserved (16 bits), Effective PSN (24 bits), and Reserved (8 bits). PARAM_ACK (Type 5) carries Config ID (8 bits), Status (8 bits; 0 accepted, 1 unsupported scheme, 2 limits exceeded, 3 resources exhausted), and Reserved (16 bits). A CSF that receives a non-zero Status keeps its current configuration.¶
REPAIR_REQUEST (Type 6) has two forms. The block form carries Config ID (8 bits), Scheme (8 bits), k (8 bits), Repairs Held r (8 bits), Block Start PSN (24 bits), Suggested M (8 bits), Request ID (16 bits), Reserved (16 bits), and the Lost Source Bitmap (ceil(k/32) words). The PSN-range form, indicated by flag 0x10 RANGES, carries Request ID (16 bits), Range Count (16 bits), and Range Count entries of Start PSN (24 bits) and Count (8 bits). Flag 0x20 FALLBACK requests Source Copy symbols for the lost sources.¶
LOSS_REPORT (Type 7) carries Interval (32 bits, microseconds), Expected (32 bits), Received (32 bits), Restored (32 bits), Incomplete Blocks (32 bits), CE Marked (32 bits), Repairs Received (32 bits), and Repairs CE Marked (32 bits), all counts for the interval.¶
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Method | Entry Count | AF | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Flow source and destination address (4 or 16 octets each) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Flow Label | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Hop Entry 1 (see below) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Hop Entry: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Node ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Marking Period Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Lost Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Queue Delay Change (signed, us) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Cause | PSN Count | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved | Dropped PSN (24 bits), repeated PSN Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Method: 1 Alternate Marking [RFC9341], 2 Clustered Alternate Marking [RFC9342], 3 node-local drop accounting. The flow is identified by addresses and Flow Label as in HELLO, because a Path Reporter does not know the Flow ID. Entry Count is at most 16.¶
Repair traffic adds load to the network. The congestion control of an RDMA flow normally runs in the RDMA interface, which a CSF in a host shim or gateway cannot influence directly, and REPAIR messages use a different UDP port and can therefore take a different path under equal-cost multipath routing. CREM therefore limits repair traffic itself:¶
Because redundancy is not raised for congestion-related loss, CREM does not amplify congestion. These rules follow [RFC8085] and [RFC9743]. As a circuit breaker in the sense of [RFC8084], a CSF that observes a path loss rate (from LOSS_REPORT, before restoration) above ten percent or a CE fraction above fifty percent over one second MUST set m to zero and stop coded retransmission for the flow, leaving recovery to the RDMA transport. The flow stays open, so LOSS_REPORT messages continue, and the CSF resumes redundancy only after the condition has been absent for at least ten seconds. If the condition persists for more than one minute, the CSF sends CLOSE with reason 4.¶
Data integrity is the main concern. A forged REPAIR message would make the CRF reconstruct wrong packets, which the RDMA receiver would place into application memory. The invariant CRC of RoCEv2 detects accidental corruption of restored packets but not a deliberate forgery. Deployments MUST therefore either protect CREM messages, and the protected flow, with IPsec ESP [RFC4303], or confine CREM to a limited domain whose boundary drops datagrams addressed to the CREM port from outside. The random Sender Nonce and Flow ID make blind injection unlikely but do not authenticate messages.¶
Forged REPAIR_REQUEST or PATH_REPORT messages could cause the CSF to send additional repair traffic. The amount is bounded: at most X symbols per block per round trip, within the rate limits of Section 10; REPAIR messages are sent only to the address bound at HELLO, never to the source of a request; and PATH_REPORT is accepted only from configured reporters. HELLO is no smaller than HELLO_ACK, so CREM cannot be used for reflection amplification.¶
Path reports reveal per-hop loss and delay. They are sent only to sources of protected flows inside the domain.¶
This document requests no port number; the CREM port is configured per limited domain (Section 3).¶
IANA is requested to create a registry group named "CREM Parameters" with the following registries. Unless stated otherwise, values are assigned by Specification Required [RFC8126] and the upper quarter of each value space is reserved for Private Use.¶
The designated experts should verify that new FEC schemes specify the coding of absent positions and whether coded retransmission is supported, and that new flow kinds specify how sources and PSN positions are identified.¶
This example uses scheme 1 with k = 4 and T = 16. The UDP payloads of the four source packets are the ASCII strings "RDMA-PKT-0" to "RDMA-PKT-3" (10 octets each), so each source symbol is the length 0x000a, the payload, and four octets of padding:¶
S0 = 000a52444d412d504b542d3000000000 S1 = 000a52444d412d504b542d3100000000 S2 = 000a52444d412d504b542d3200000000 S3 = 000a52444d412d504b542d3300000000¶
Coefficients C(i, j) for i = 0..2 and j = 0..3 (x_i = 255 - i, y_j = j):¶
j=0 j=1 j=2 j=3
i=0: fd 7e ff 7f
i=1: 7e fd 7f ff
i=2: ff 7f fd 7e
¶
With m = 2 the CSF sends R0 and R1 over the whole block:¶
R0 = 001ef6ccd7c377f0ddfc774c00000000 R1 = 001ef6ccd7c377f0ddfc774f00000000¶
Sources 0, 1, and 2 are lost (X = 3); the CRF holds S3, R0, and R1 (r = 2) and requests M = X - r = 1 symbol for L = {0, 1, 2}. The CSF sends the coded retransmission with Repair Index 2 over L only:¶
R2[L] = 00354fdf6d4b40b57e5c400000000000¶
The CRF subtracts C(0,3)*S3 from R0 and C(1,3)*S3 from R1 and solves the 3 x 3 Cauchy system with rows i = 0, 1, 2 and columns j = 0, 1, 2, recovering S0, S1, and S2. Three lost packets were recovered with one retransmitted symbol. The REPAIR message for R2[L] (Flow ID 0x0000BEEF, Config ID 1, Block Start PSN 0x001000, Repair Session ID 7, all four positions present, so Present Bitmap 0xF0000000 and Lost Source Bitmap 0xE0000000) begins as follows and is followed by the 16 octets of R2[L], for a total Length of 44:¶
11 80 00 2c 00 00 be ef 01 01 04 02 00 10 00 00 00 07 00 10 f0 00 00 00 e0 00 00 00¶
The authors thank the participants in discussions on RDMA over long-distance networks for their comments.¶