China Mobile

Beijing 100053 China yangjinwl@chinamobile.com

China Mobile

Beijing 100053 China tianyuchi@chinamobile.com

China Mobile

Beijing 100053 China chengweiqiang@chinamobile.com

China Mobile

Beijing 100053 China hantingting@chinamobile.com

Centec

Suzhou 215000 China wangjj@centec.com

Centec

Suzhou 215000 China zhanggy@centec.com

RTG RTGWG ECMP load balancing flow placement next-hop selection LAG Equal-Cost Multipath (ECMP) forwarding assigns a flow to a next-hop by computing a hash over packet header fields when the flow is first seen. Because the assignment does not change after that, the load across the paths of an ECMP group can become uneven when the set of paths changes or when traffic shifts among existing flows. This document defines procedures for adaptive ECMP load balancing in which a forwarding node periodically measures per-path load, determines whether the load is sufficiently uneven to act on, and makes a bounded number of incremental flow placement changes toward less-used paths. The procedures preserve per-flow packet ordering for all but the small number of flows that are moved in any interval, and they converge toward a balanced distribution across the group. The procedures operate within a single forwarding node and add no new packet formats or inter-node signaling.

Introduction Equal-Cost Multipath (ECMP) routing lets a forwarding node distribute IP traffic over several next-hops of equal routing cost. The forwarding decision is commonly made by hashing selected packet header fields and mapping each flow to one path. analyzes the hash-threshold algorithm that is widely used for this purpose and discusses the trade-off between distribution uniformity and the disruption that occurs when paths are added or removed. Hash-based path selection gives a reasonable initial distribution and preserves per-flow packet ordering, but the assignment is fixed: once a flow is mapped to a path it stays there regardless of later changes to the group or to traffic. As a result the load across the paths of a group can become uneven over time. This is a known operational effect. describes techniques for identifying large ("elephant") flows and selectively moving them across LAG or ECMP component links to improve utilization. That document is Informational, addresses primarily the classification of large flows, and does not define normative requirements for general load-aware path selection. This document defines normative procedures for load-aware ECMP path selection that apply to all flows in an ECMP group, not only to large flows. The procedures rest on three elements: periodic measurement of per-path load, a bounded number of flow placement changes per interval to limit the rate of path changes, and operator-controlled parameters that govern the behavior. The procedures supplement, and do not replace, the existing hash-based path selection: they add a load-aware feedback loop on top of the hash function. The procedures carry MUST-level requirements on forwarding-node behavior and are therefore on the Standards Track. The relationship of these procedures to entropy-improving mechanisms and to other load-balancing work is described in .

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Problem Statement Hash-based ECMP next-hop selection gives a one-time, deterministic mapping from a flow identifier to a path. The mapping does not account for runtime conditions, which leads to uneven load in at least the following cases.

Path Addition

When a new equal-cost next-hop becomes available, for example when a link comes up or a route is installed, only newly arriving flows are candidates for the new path. Existing flows stay on their assigned paths. The new path can remain under-used until enough flows turn over naturally, which takes an unpredictable time that depends on flow lifetimes.

Asymmetric Flow Departure

When flows end unevenly across paths, for example when several long-lived flows on one path end while flows on other paths continue, the load becomes uneven. The capacity freed on the path with departing flows is not taken up by flows on the more heavily used paths.

Traffic Rate Variation

Even when the set of active flows is stable, changes in per-flow rate, for example a flow entering a bulk-transfer phase, can make the load on one path diverge from the others. A static hash assignment does not respond to such rate changes.

In each case the forwarding node has enough local information, in the form of per-path counters and flow-table state, to detect the imbalance, but a hash-only ECMP implementation does not act on it. This document specifies how a forwarding node uses locally available load data to make incremental changes to flow placement.

Terminology This document uses the following terms. Where applicable, the definitions of and are reused.

ECMP Group

A set of equal-cost next-hops toward the same destination prefix, among which traffic is forwarded. A Link Aggregation Group (LAG) is treated as the logical equivalent of an ECMP group in this document.

Path (Component Link)

An individual next-hop within an ECMP group. The term "component link" is used interchangeably, consistent with .

Flow

A sequence of packets that share common header fields and are treated as a unit for path selection. The flow granularity, for example a 5-tuple or a 3-tuple, is set by the hashing policy and is outside the scope of this document.

Load Metric

A quantitative measure of the traffic carried by a path over a measurement window, for example a forwarded byte count, a forwarded packet count, or a count of active flows.

Imbalance Ratio (IR)

A dimensionless measure of the degree of load skew across the paths of an ECMP group, defined in .

Reassignment Limit

An upper bound on the number of flow placement changes that may be made within one evaluation interval. It is derived from the observed imbalance and controls the rate of change.

Evaluation Interval

The periodic window at which the forwarding node collects load metrics, assesses imbalance, and determines whether placement changes are warranted.

Design Objectives The procedures defined in this document follow these objectives.

Minimal Disruption

The number of flows whose paths change in any interval MUST be bounded. Unbounded path changes would cause widespread packet reordering and degrade transport performance .

Proportional Response

The number of flow placement changes SHOULD be proportional to the severity of the imbalance. When the load is within bounds, no changes are made; when the imbalance is large, more changes are permitted per interval to converge faster.

Incremental Convergence

Load balancing SHOULD converge over successive intervals rather than attempting to reach a balanced state in a single step. Incremental convergence avoids oscillation and gives stable behavior.

Operational Control

All parameters, including thresholds, limits, intervals, and eligible flow classes, MUST be configurable by the operator. The procedures MUST be able to be disabled on a per-group basis without affecting other groups.

High-Level Operation No action; wait for +------+------+ next interval | no v +---------------------+ | Derive reassignment | | limit from IR | +----------+----------+ | v +---------------------+ | Adjust flow | | placement (up to | | limit) on eligible | | events | +---------------------+ ]]>

Load Measurement

Load Metrics A forwarding node that implements these procedures MUST maintain per-path load metrics for each ECMP group on which adaptive load balancing is enabled. The node MUST support at least one of the following metric types.

• Forwarded byte count over the evaluation interval. This reflects bandwidth use and is RECOMMENDED as the default.
• Forwarded packet count over the evaluation interval.
• Number of distinct active flows observed during the evaluation interval.

A node MAY support a weighted combination of metrics or additional implementation-defined metrics. The active metric type SHOULD be configurable per group. The evaluation interval MUST be configurable with a granularity of at least 100 milliseconds. The default interval SHOULD be 1 second. An interval shorter than 100 milliseconds is NOT RECOMMENDED because of the processing cost of frequent measurement.

Imbalance Assessment At each evaluation interval the forwarding node computes the imbalance ratio (IR) across the N paths of the ECMP group. Let U_i be the measured load metric for path i, where 1 <= i <= N. 0 IR = 0, when U_avg = 0 ]]> The IR is compared with a configurable imbalance threshold (T_imb). When IR is below T_imb, the load is considered acceptable and no flow placement change is made during the interval. The default T_imb SHOULD be 0.3, which corresponds to a 30 percent normalized deviation. An operator MAY change this value: a lower threshold reacts to smaller imbalances at the cost of more frequent changes, and a higher threshold reduces the frequency of changes at the cost of tolerating more skew.

Reassignment Limit Derivation When IR >= T_imb, the forwarding node derives a reassignment limit (R_lim) that bounds the number of flow placement changes made during the interval. The derivation of R_lim MUST satisfy the following properties.

Non-decreasing

R_lim MUST be a non-decreasing function of IR, so that a larger imbalance permits more changes per interval.

Bounded

R_lim MUST NOT exceed a configurable upper bound (R_max), which prevents excessive changes under extreme conditions. The default R_max SHOULD be proportional to the number of active flows in the group, with a RECOMMENDED default of min(F/10, 16), where F is the number of active flows. A node MAY use a fixed default, for example 8, when the active flow count is not readily available.

Zero Below Threshold

When IR < T_imb, R_lim MUST be zero.

The function used to map IR to R_lim is an implementation choice. Linear, piecewise-linear, and step functions are all acceptable, provided the properties above hold. The following is a non-normative example of a compliant function. = T_imb where S is a configurable scaling factor (default: 10). ]]> This example is given for clarity only. An implementation may use any function that satisfies the stated properties.

Flow Placement Adjustment

Eligible Events Flow placement changes are made in response to specific forwarding-plane events, subject to the reassignment limit for the interval. Each change consumes one unit of the reassignment limit. The following event types are defined.

New Flow Arrival

On the first packet of a previously unseen flow, the node MAY assign the flow to the least-used path rather than to the path given by the hash function. A node MUST support this event type.

Flow Expiration

When a flow entry ages out or is cleared, the resulting change in per-path load MAY trigger the reassignment of another flow from a more heavily used path to a less used path. Support for this event type is RECOMMENDED.

Periodic Sweep

A background process MAY iterate over local flow state and select flows on the most-used path for reassignment to the least-used path. Support for this event type is OPTIONAL.

Operator-Initiated

A management action MAY request that the node make flow placement changes, for example during a maintenance window. Support for this event type is OPTIONAL.

The set of active event types SHOULD be configurable. When more than one event type is enabled, any of them may trigger a change while the reassignment limit for the interval has not been reached.

Target Path Selection When a flow placement change is made, the target path MUST be the path with the lowest current load metric in the group at the time of the decision. When more than one path shares the lowest load, the node MAY choose any of them, for example by a secondary hash or round-robin among the candidates. When a change is triggered by a periodic sweep or a flow expiration, the source path from which a flow is selected MUST be the path with the highest current load metric. The choice of which flow to move from the source path is an implementation decision. A node SHOULD prefer flows with lower instantaneous rate, to limit the effect of reordering on any single flow.

Rate of Path Changes The reassignment limit is the primary means of controlling the rate of flow placement changes. Each change reduces the remaining limit by one for the interval. When the remaining limit reaches zero, no further changes are made until the next interval begins with a freshly derived limit. A node MUST NOT carry over unused reassignment capacity from one interval to the next. The limit is derived anew at each interval from the load observed for that interval. This prevents the accumulation of unused capacity that could otherwise cause a burst of path changes. The combination of a bounded limit and periodic evaluation forms a self-regulating feedback loop. Over successive intervals the per-path load converges toward the threshold. Once IR falls below T_imb, the limit becomes zero and changes stop until conditions change.

Interaction with Existing Forwarding Behavior

Coexistence with Hash-Based Path Selection These procedures supplement, and do not replace, the existing hash-based ECMP next-hop selection. The hash function remains the default path selection method for all flows. A load-aware placement change overrides the hash-based assignment only for a specific flow and only when the derived reassignment limit permits it. When adaptive load balancing is disabled, or when the reassignment limit is zero, the node MUST use the standard hash-based path selection for all flows.

Applicability to Link Aggregation Groups A LAG selects a member link by a hash in a manner analogous to ECMP next-hop selection. These procedures apply equally to LAG member selection. When applied to a LAG, the references in this document to "ECMP group" and "path" apply to the LAG and its component links respectively. A node that implements adaptive load balancing for both ECMP groups and LAGs SHOULD manage them independently, since each has its own membership, load, and parameters.

MPLS Environments In MPLS environments, entropy labels add input to the hash and improve the uniformity of the initial flow placement. Even with entropy labels, the fixed nature of the hash assignment means that imbalance can develop over time, so these procedures remain applicable as a complementary technique.

Relationship to Other Work The procedures in this document make the path-selection decision load-aware. A separate class of mechanisms instead improves the entropy that is fed into a static hash, so that the initial distribution is more uniform; examples are MPLS entropy labels , the IPv6 flow label , and, for IPsec ESP traffic, the ESP Entropy Header . These entropy-improving mechanisms and the load-aware procedures of this document address different aspects of the same problem and are complementary: better entropy improves the initial distribution, while load-aware placement corrects the skew that develops after flows are assigned. Neither removes the need for the other. also addresses adaptive load balancing for ECMP, with an emphasis on flowlet-based methods and the difficulty of configuring a flowlet timeout. The procedures in this document differ in that they operate as a node-local feedback loop with an explicit imbalance assessment and a bounded, proportional number of flow placement changes per interval, and they are not tied to flowlet detection. The two approaches may inform each other; this document does not depend on flowlet detection.

Applicability Considerations

Data Center Fabrics Data center leaf-spine fabrics commonly use large numbers of ECMP paths . The combination of high path count, a mix of flow sizes including long-lived storage and compute flows, and high utilization targets makes these environments prone to hash-induced load skew. These procedures can improve fabric-wide load balancing without a central controller.

Wide Area Networks In WAN environments where ECMP is used across parallel links or tunnels, the smaller number of concurrent flows and their longer lifetimes increase the effect of each flow placement decision on the aggregate load. An operator that deploys these procedures in a WAN SHOULD consider a longer evaluation interval and a lower reassignment limit, to reflect the higher cost of a path change in a wide-area topology.

Operational Considerations

Configuration Parameters The following parameters govern the behavior described in this document. A node MUST allow these parameters to be configured per ECMP group, or per LAG. Configuration Parameters

Parameter	Description	Default
Enable/Disable	Activate or deactivate adaptive load balancing for the group	Disabled
Evaluation Interval	Time between consecutive load assessments	1 second
Load Metric Type	Metric used for per-path comparison	Byte count
Imbalance Threshold (T_imb)	IR below which no change is made	0.3
Maximum Reassignment Limit (R_max)	Upper bound on flow placement changes per interval	8
Active Event Types	Which events may trigger a change	New flow arrival

A parameter change SHOULD take effect at the start of the next evaluation interval, to avoid inconsistent behavior during a running interval. Disabling the feature MUST cause the node to stop making load-aware changes immediately and to revert to hash-based path selection for subsequent flows. Flows that were previously moved MAY remain on their current paths; the node is not required to return them to their hash-computed paths on disablement.

Flow Exclusion Policies Some traffic classes require path persistence and MUST NOT be subject to flow placement change. A node MUST support the configuration of exclusion policies that identify flows exempt from change. Examples of traffic that requires exclusion are the following.

• Flows that traverse stateful middleboxes such as firewalls, NAT gateways, or intrusion prevention systems, where a path change would invalidate session state.
• Real-time media flows, such as voice or video, that are sensitive to jitter and reordering.
• Flows marked with a DSCP value that indicates a path affinity requirement.

An exclusion policy SHOULD support matching on standard packet header fields, including source and destination address prefixes, protocol numbers, port ranges, and DSCP values. An excluded flow MUST NOT consume reassignment capacity and MUST remain on its hash-assigned path.

Monitoring A node SHOULD expose the following operational state for each ECMP group on which adaptive load balancing is enabled.

• Current per-path load metric values.
• Most recently computed imbalance ratio.
• Reassignment limit derived for the current interval and the portion consumed.
• Cumulative count of flow placement changes since the feature was enabled.
• Number of flows currently excluded by policy.

This state SHOULD be accessible through standard management interfaces, such as a YANG-based management protocol , SNMP, or streaming telemetry. This document does not define a data model.

Deployment Guidance An operator is advised to adopt a conservative initial posture.

• Enable the feature on a small number of ECMP groups first and observe the imbalance ratio and change counts before wider rollout.
• Use a higher imbalance threshold, for example 0.5, and a lower reassignment limit initially, then tighten the parameters as confidence grows.
• Ensure that all stateful middlebox traversals are covered by exclusion policies before enabling the feature.

Conformance and Interoperability The procedures in this document constrain the forwarding-plane behavior of a single node and do not define any new packet format, header field, or inter-node protocol. The packets a conforming node emits are bit-for-bit indistinguishable from those of a node that performs only hash-based path selection; a flow placement change alters which member of an ECMP group or LAG a flow egresses on, not the contents of the flow's packets. Interoperability with neighbors, with transport endpoints, and with existing ECMP and LAG deployments is therefore preserved unconditionally. The normative requirements in this document are stated as externally observable properties so that conformance can be verified without access to a node's internal implementation. In particular, a conforming implementation:

• changes the path of no more than the derived reassignment limit () of flows in any evaluation interval, and makes no changes while the imbalance ratio is below the configured threshold ();
• preserves per-flow ordering for every flow that is not moved in a given interval, and moves a flow at most as a single, atomic change of egress member;
• never moves a flow that matches a configured exclusion policy ();
• reverts to hash-based path selection for subsequent flows when the feature is disabled (); and
• exposes the operational state of through which the preceding properties can be observed.

These properties are testable at a node's interfaces: an operator or test harness can offer a controlled traffic pattern, observe per-path load and the reported imbalance ratio, and confirm that the number and rate of flow movements respect the configured bounds and exclusions. The normative requirements thus give operators an enforceable basis for procurement and conformance testing, which is the reason this document is placed on the Standards Track even though it introduces no on-the-wire protocol element.

Security Considerations The procedures in this document operate within the local forwarding plane of a single node. They introduce no new protocol messages, packet formats, or inter-node signaling, and so do not expand the control-plane attack surface. The following data-plane considerations apply.

Stateful Middlebox Bypass

A flow placement change causes a flow's packets to use a different path, which may bypass a stateful inspection device. An operator MUST configure exclusion policies () for all flows subject to stateful processing.

Induced Path Changes

An attacker that can inject traffic with chosen flow identifiers could try to concentrate load on one path and cause repeated changes. The reassignment limit bounds the per-interval cost of such an attempt, the imbalance threshold prevents reaction to minor fluctuations, and the evaluation interval imposes a minimum time between rounds of changes. Together these bound the amplification.

Flow State Resource Consumption

A node that keeps per-flow state to support placement changes, for example to record which flows have been moved, SHOULD bound the size of that state to prevent memory exhaustion, especially where flow counts are high.

IANA Considerations This document has no IANA actions.

References Normative References Informative References

Acknowledgements The authors thank the members of the IETF Routing Area Working Group for their review and feedback.