]> Fastly

kazuhooku@gmail.com

Congestion Control Working Group internet-draft This document specifies Cuback, an ACK-driven reformulation of CUBIC congestion control. It simplifies implementation by replacing CUBIC's mutable time- and ACK-driven state with pure functions over immutable per-epoch parameters, for which test vectors are provided. Congestion-window growth uses the same ACK-driven mechanism as Reno, removing several sources of implementation error. When entering a high-capacity path, a newcomer converges on its share sooner than under CUBIC, so short flows complete earlier. Discussion Venues Discussion of this document takes place on the Congestion Control Working Group Working Group mailing list (ccwg@ietf.org), which is archived at . Source for this draft and an issue tracker can be found at .

Introduction CUBIC is a widely deployed congestion-control algorithm that improves scalability over Reno by defining congestion-window growth as a cubic function of elapsed time. Its implementation, however, is more complex than the growth function itself might suggest. During congestion avoidance, a CUBIC sender maintains two independently evolving window estimates: the time-driven cubic window and the ACK-driven Reno-friendly window. The sender must update both estimates and select the larger of the two. In addition, because the cubic window advances with wall-clock time, the sender needs mutable epoch state and a state machine that pauses the cubic clock when the sender ceases to be congestion-control limited and resumes it when transmission becomes limited by the congestion window again. Correctly maintaining these states complicates implementations and their interaction with application-limited periods and other congestion-control transitions. This document specifies Cuback, an ACK-driven reformulation of CUBIC. Cuback expresses both the cubic and Reno-friendly growth functions in terms of the amount of data that must be acknowledged for the congestion window to reach a given value. The parameters defining those functions are established at the beginning of a congestion-avoidance epoch and remain immutable for the duration of that epoch. Congestion-window growth can therefore be calculated from a pure function of the current congestion window and the per-epoch parameters, without maintaining separately evolving cubic and Reno-friendly window estimates. This formulation also eliminates the need to pause and resume a wall-clock-driven CUBIC epoch. When acknowledgments stop arriving, advancement along the Cuback curves stops naturally. The remaining runtime state is essentially the same as in Reno: the sender tracks how many additional bytes need to be acknowledged before increasing the congestion window by one MSS. Cuback therefore retains the characteristic growth behavior of CUBIC while reducing congestion avoidance to the same ACK-driven window-increase mechanism used by Reno. In both Reno and Cuback, acknowledged bytes are accumulated until enough have been received to increase the congestion window by one MSS. The difference lies only in how the required number of acknowledged bytes is calculated: Reno derives it directly from the current congestion window, whereas Cuback obtains it by evaluating the pure functions defined in this document. What remains of CUBIC's complexity is confined to a pure function of the congestion window and the per-epoch immutables, which can be validated directly from its inputs and outputs without exercising a sequence of state-machine transitions. An implementation can therefore be checked against known values, and is correspondingly less prone to error. This document includes test vectors for that purpose. The use of the ACK clock has a performance benefit as well. When a new flow joins a high-capacity path already carrying an established flow, it acquires its share more rapidly than it would under CUBIC. Cuback derives its growth rate from the congestion window and round-trip time recorded at the previous congestion event. Because the cubic function gives a flow with a smaller window a larger proportional increase, a newcomer gains share; its ACK rate, and hence its ACK clock, runs faster than originally anticipated. This gain compounds over successive round trips, and reduces the time to completion of short flows, such as the delivery of HTTP objects. The advantage diminishes as the flow's rate converges on the value it recorded. Cuback alters only the congestion avoidance stage of CUBIC. All other behavior specified in applies unchanged.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document uses the notation of , in which window sizes are expressed in segments. An implementation that counts bytes instead scales each window value by SMSS, divides the sums appearing in by SMSS, and expresses the lower bound on D(w) as 2 * SMSS.

Cuback Congestion Avoidance During congestion avoidance, a Reno sender accumulates the data acknowledged and increases cwnd by one segment each time a threshold D(cwnd) is reached: = ssthresh: pending = pending + segments_acked while pending >= D(cwnd): pending = pending - D(cwnd) cwnd = cwnd + 1 ]]> In Reno, that threshold is the congestion window itself: Cuback retains the mechanism above unchanged and replaces only the threshold, where A(w) is the amount of data, counted from the beginning of the current congestion avoidance stage, that has to be acknowledged for the congestion window to reach w: W_max K(W_max, cwnd_epoch) = cbrt((W_max - cwnd_epoch) / C) ]]> W_max, bandwidth, and cwnd_epoch are fixed when the congestion avoidance stage begins and do not change until the next congestion event:

• W_max is as defined in . It is set to cwnd_prior, the congestion window before the reduction at the congestion event that began the stage.
• bandwidth is an estimate of the rate at which the bottleneck delivers data, in segments per second. It is set at the congestion event to cwnd_prior divided by the smoothed round-trip time.
• cwnd_epoch is the congestion window at the beginning of the congestion avoidance stage, as defined in .

alpha_cubic, beta_cubic, and C are the constants defined in , with recommended values: A sender that derives ssthresh from flight_size rather than from cwnd () MUST use flight_size in place of cwnd_prior throughout this section. W_max, cwnd_epoch, and bandwidth MUST be derived from the same quantity. When congestion avoidance is entered without a congestion event, such as on exiting slow start, the parameters are established at that point instead. cwnd is not reduced, so W_max equals cwnd_epoch and K is zero, and A(w) describes growth along the convex region alone. The expressions above are derived in .

Fast Convergence Fast convergence is applied as described in : on a congestion event, if cwnd is below W_max, W_max is reduced before cwnd is: keys the switch to alpha_cubic = 1 to cwnd_prior rather than to W_max, so A_reno in uses cwnd_prior in its place; a sender that does not retain it can recover it as cwnd_epoch / beta_cubic.

Spurious Congestion Events When a congestion event is determined to have been spurious (), a sender that reverts cwnd and ssthresh MUST also restore W_max, bandwidth, and cwnd_epoch to the values they held before the event.

Application-Limited Senders places two separate requirements on an application-limited sender. requires that the elapsed time used by the cubic curve exclude application-limited periods, and restricts the increase of cwnd. The former has no counterpart in Cuback, there being no clock to pause and resume. Growth is governed by acknowledged data rather than by elapsed time: an idle sender accrues no growth, while a sender that continues to deliver data below cwnd grows only in proportion to that delivered data. The latter applies to Cuback as it does to . Stacks might meet it by applying as described in . Other means are also possible, such as suspending the increase while bytes in flight are below cwnd, as in .

Applying Rate-Limited Increase caps cwnd at limit(maxFS), the largest congestion window that the congestion controller would yield from the acknowledgements produced by one window of maxFS, the largest FlightSize observed since cwnd was last reduced. Each congestion controller defines its own limit(). Under , the growth produced by a window of acknowledgements results from per-acknowledgement updates to cwnd and W_est, which depend on each other and on the time at which each acknowledgement arrives, making limit() complex to derive. Under Cuback, it is determined by A(w) alone. Within one window of acknowledgements, cwnd can be raised from maxFS to w + 1 approximately when A(w + 1) - A(maxFS) does not exceed maxFS, with the lower bound on D(w) further limiting cwnd to 1.5 * maxFS. Separately, as in the limit that gives for Reno, cwnd is allowed to reach maxFS + 1, so that a sender limited by cwnd continues to grow even where a one-segment increase takes more than one window of acknowledgements. An increase of cwnd from w to w + 1 is therefore permitted when: A(w + 1) is evaluated for D(w) in any case, and A(maxFS) changes only when maxFS does, so the check adds no cube root to an increase.

Relationship to RFC 9438

Sensitivity to the Bandwidth Estimate bandwidth scales A_cubic linearly, so an error in it is an error in the rate at which the cubic curve is traversed: an underestimate traverses the curve more quickly than elapsed time would, and an overestimate more slowly. The round-trip time it is derived from has to describe the path as it was while cwnd was at its peak, with the bottleneck queue at its deepest. The error is bounded in either direction by mechanisms already present. An overstated bandwidth makes A_cubic large, and A_reno carries no bandwidth term at all, so A(w) becomes A_reno(w): the flow then grows at the Reno-friendly rate, and no error in bandwidth can slow it below that. An understated bandwidth makes A_cubic small, and the lower bound of two segments on D(w) caps growth at half the congestion window per round trip, which adopts to keep the increase below that of slow start. Between those bounds the estimate matters only at windows large enough for the cubic curve to govern. Windows that large deliver many acknowledgements per round trip, and a QUIC sender takes a round-trip sample from every acknowledgement that advances the largest acknowledged packet number , so the smoothed round-trip time follows the path closely by the time the estimate matters.

Convergence under the ACK Clock Deriving the clock from acknowledgements reinforces the convergence that AIMD provides, because bandwidth records the rate the flow was achieving before it reduced. A flow holding a large share gives up the most at a congestion event, so the rate it goes on to achieve falls below the value it recorded and its clock runs slow. A flow arriving with a small share records a correspondingly small rate and then achieves more than it recorded, so its clock runs fast. The congestion window advances fastest for the flow gaining share and slowest for the one giving it up. The curve advances at the ratio of the rate the flow is currently achieving to the rate recorded at the last congestion event. A flow that has since doubled its share traverses the curve twice as fast as elapsed time would carry it; one that has halved its share, half as fast. The advantage therefore decays as the achieved rate converges on the recorded one.

Sensitivity to the Timing of Clock Transitions requires that elapsed time exclude periods during which cwnd was not updated because the sender was application limited. A CUBIC sender therefore has to pause its clock as the flow becomes application limited and resume it as the flow ceases to be, and an error at either edge is an error in the elapsed time the curve is read against. Treating an application-limited sender as congestion-window limited leaves the clock running. W_cubic(t) advances while the flow is not filling cwnd, and cwnd grows on evidence the path never supplied; names the consequence, that W_cubic(t) "might be very high after restarting from these periods". Treating a congestion-window-limited sender as application limited stops the clock instead, so the elapsed time that should have advanced the curve is never accrued and the flow grows too slowly. Neither edge is easy to observe. A sender might emit as many datagrams as it can in one pass and so appear limited by the congestion controller, even though its bytes in flight are below cwnd. Conversely, an in-stack pacer might withhold emission after a train of acknowledgements, leaving bytes in flight below cwnd although the connection is still limited by the congestion window. updates , mitigating the impact of such overgrowth by capping cwnd at a value derived from the largest FlightSize observed, but a CUBIC sender is still required to pause and resume its clock at the correct moments. Cuback has no clock to stop or start. When the sender stops, the advance of A(w) stops with it: no acknowledgements arrive, and nothing accrues to be caught up when transmission resumes. The case in which a mistimed CUBIC clock does the most damage therefore does not arise. A sender that keeps sending but holds less than cwnd in flight is not protected altogether, only bounded. Growth is governed by the amount of data acknowledged, as in Reno, so cwnd advances in proportion to what the path delivered rather than in proportion to elapsed time. Applying caps what remains, holding cwnd to what one window of the largest FlightSize observed would yield.

The Reno-Friendly Estimate advances W_est on each acknowledgement: where cwnd is the congestion window in effect. While the cubic curve governs, that window is larger than W_est, so the growth of W_est depends on the trajectory of the other curve. This coupling cannot be expressed as a function of W_est alone, and A_reno instead inverts the standalone recurrence, in which the divisor is the Reno-friendly window itself. Where A_reno is the smaller of the two curves, cwnd is the Reno-friendly window and the two formulations coincide exactly: both increase the window by one segment for every cwnd / alpha_cubic segments acknowledged. They differ only in where that region is entered. Because the standalone recurrence uses the smaller divisor, it advances faster while the cubic curve governs, and a Cuback sender enters the Reno-friendly region somewhat earlier than a sender following .

Smoothing the Cubic Growth and advance cwnd toward a target one round trip ahead, W_cubic(t + RTT), rather than assigning W_cubic(t) to cwnd directly. Because the CUBIC curve is driven by elapsed time, the growth owed at an acknowledgement depends on how long has passed since the previous one, and assigning the curve directly would turn a gap in acknowledgements into a step in cwnd, which is a burst. The one-round-trip target, together with the per-acknowledgement increase of (target - cwnd) / cwnd, spreads one round trip of the curve across one round trip of acknowledgements, making the increase proportional to the data acknowledged however the acknowledgements are distributed in time. Cuback needs no such smoothing. D(w) is a function of acknowledged data alone, so a gap in acknowledgements accrues no growth to be caught up, and the increase is proportional to the ack stream by construction.

Calculation Overhead A_reno is arithmetic, but A_cubic requires a cube root. evaluates one when the epoch begins, to obtain K, and thereafter only cubes; Cuback evaluates one whenever it computes D(w). Storing the data still needed to increase the window by one segment, rather than recomputing D(w) on every acknowledgement, moves the evaluation from once per acknowledgement to once per increase. Each increase then evaluates three roots: K, A_cubic(w), and A_cubic(w + 1). Calculating a cube root to the accuracy required here is cheap. The approximation must be continuous in slope, adjacent evaluations being differenced, and one meeting that condition is given in . It runs in about nine cycles on contemporary hardware, against roughly nine hundred to encrypt a 1460-byte segment through an AES-GCM-128 pipeline at 1.6 bytes per cycle. In the worst case, the window growing by one segment for every segment acknowledged, the overhead is therefore around three percent of the cost of encrypting the data that drove it; during congestion avoidance the window typically grows far more slowly than that. Furthermore, K is fixed for the duration of the epoch and could be retained rather than recomputed, and A(w + 1) for one increase is A(w) for the next and could be carried forward. Together these leave a single cube root per increase.

Security Considerations TODO Security

IANA Considerations This document has no IANA actions.

References Normative References CUBIC is a standard TCP congestion control algorithm that uses a cubic function instead of a linear congestion window increase function to improve scalability and stability over fast and long-distance networks. CUBIC has been adopted as the default TCP congestion control algorithm by the Linux, Windows, and Apple stacks. This document updates the specification of CUBIC to include algorithmic improvements based on these implementations and recent academic work. Based on the extensive deployment experience with CUBIC, this document also moves the specification to the Standards Track and obsoletes RFC 8312. This document also updates RFC 5681, to allow for CUBIC's occasionally more aggressive sending behavior. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References This document defines TCP's four intertwined congestion control algorithms: slow start, congestion avoidance, fast retransmit, and fast recovery. In addition, the document specifies how TCP should begin transmission after a relatively long idle period, as well as discussing various acknowledgment generation methods. This document obsoletes RFC 2581. [STANDARDS-TRACK] University of Oslo University of Washington University of Aberdeen National Institute of Technology Karnataka This document specifies how transport protocols increase their congestion window when the sender is rate-limited, and updates RFCs 4341, 5681, 9002, 9260, and 9438. Such a limitation can be caused by the sending application not supplying data or by receiver flow control. This document describes loss detection and congestion control mechanisms for QUIC.

Derivation of the Increase Function The lower bound of two segments in D(w) limits the increase to one half of the congestion window per round trip: cwnd segments are acknowledged per round trip, and each one-segment increase consumes at least two of them. It is the counterpart of the constraint target <= 1.5 * cwnd in and . No lower bound on the increase is needed, as A(w) is monotonically increasing and D(w) is therefore always positive. CUBIC sets its congestion window to the larger of two window growth functions. Both are monotonically increasing, so their maximum is monotonically increasing as well, and the inverse of that maximum is the minimum of the two inverses. A(w) is therefore obtained by inverting each function separately and taking the smaller of the two results. A_cubic is the inverse of W_cubic(t) of , multiplied by bandwidth to express elapsed time as acknowledged data. That the curve is zero at w = cwnd_epoch follows from the definition of K. Sampling bandwidth as cwnd_prior / RTT is the rate at which the flow was delivering data over the round trip preceding the congestion event. It carries an identity: While the congestion window is at cwnd_prior, one round trip of acknowledgements advances A(w) by exactly one round trip along the underlying cubic curve. A_reno is the inverse of the Reno-friendly increase of : one segment for every cwnd / alpha_cubic segments acknowledged, with alpha_cubic replaced by one once the window reaches W_max. Summing those thresholds over the segment-spaced windows from cwnd_epoch to w yields the expression in . Differencing A_reno recovers exactly the threshold from which it was built: The Reno-friendly region is therefore not a separate mechanism in Cuback. It is the value that D(w) takes wherever A_reno is the smaller of the two curves. The closed form exists only so that the two curves can be compared. Differencing A_cubic eliminates K, so an implementation need not evaluate it when the cubic curve is the smaller of the two:

Test Vectors A(w) and D(w) are pure functions of the congestion window and the parameters of , so they can be checked directly. Both vectors below use the recommended constants, with alpha_cubic taken as 3 * (1 - beta_cubic) / (1 + beta_cubic) rather than the rounded 0.529. The first exercises the Reno-friendly curve. bandwidth is large enough here that A_cubic exceeds A_reno at every window shown, so A(w) is A_reno(w) throughout. The window reaches cwnd_prior at 10 segments, where alpha_cubic is replaced by one and each further segment costs exactly its own window in acknowledged data. The second exercises the cubic curve, the parameters being chosen so that K is exactly 10 seconds. A_reno exceeds A_cubic at every window shown, so A(w) is A_cubic(w) throughout. A(W_max) divided by bandwidth recovers K, and the data needed to climb the 50 segments below W_max equals that needed for the 50 above, the curve being point-symmetric about W_max. D(w) on the cubic curve is the difference of two cube roots taken at adjacent windows, so an implementation using an approximation should compare accumulated values, as above, rather than individual increases; see .

An Approximation of the Cube Root The cube root function below is provided for reference and is placed in the public domain. It assumes IEEE 754 binary64. Its maximum relative error is about 3.5e-5 and the relative error of its slope is below 8.5e-4, holding each per-segment increase to roughly 0.1 percent, and it runs in about nine cycles on a Zen 3 core. > 52; if (abs_u == 0) return x; if (exp == 0 || exp == 0x7ff) return cbrt(x); int e = (int)exp - 1023; int q = e / 3; int r = e - 3 * q; if (r < 0) { r += 3; --q; } uint64_t mant = abs_u & 0xfffffffffffff; double m = BITS2DBL(mant | ((uint64_t)1023 << 52)); double y = -0.010279630422175424; y = 0.08469719299100167 + m * y; y = -0.2988437245358902 + m * y; y = 0.7177663288981584 + m * y; y = 0.5066598330689034 + m * y; static const double cbrt2_to_r[3] = {1.0, 1.2599210498948731648, 1.5874010519681994748}; double two_to_q = BITS2DBL((uint64_t)(q + 1023) << 52); y *= cbrt2_to_r[r] * two_to_q; return (u >> 63) ? -y : y; } ]]>

Acknowledgments TODO acknowledge.