Latency Guarantee with Stateless Fair Queuing

Recommendation ITU-T Y.3129 specifies the requirements and framework for Deterministic Networking (DetNet) using a set of work-conserving packet schedulers that guarantee end-to-end (E2E) latency bounds to flows. Recommendation ITU-T Y.3148 further specifies the functional architecture, functional entities, operational procedures, and packet metadata required to fulfill the requirements and framework specified in Y.3129. In this framework, core nodes do not maintain per-flow state. The edge node through which a flow enters the network, referred to as the entrance node, computes for each packet the ideal service completion time under a fluid model, referred to as the Finish Time (FT), and records it in the packet header. Each subsequent core node updates the FT, without per-flow state, by adding a delay factor that is a function of the parameters of the flow and of the upstream nodes. The scheduler serves queued packets in ascending order of FT. This mechanism is referred to as stateless fair queuing, and the resulting framework is referred to as Work-conserving Stateless Core Fair Queuing (C-SCORE) . C-SCORE provides strong isolation among flows. For a certain choice of the delay factor, a closed-form expression for the E2E latency bound of a flow can be derived. This bound is identical to that of a network in which every node employs a stateful fair queuing scheduler. It depends on the flow's intrinsic parameters, such as its maximum burst size and service rate, and on link parameters, such as the link capacity and the maximum packet length among the other flows sharing each output link with the flow. The key component of C-SCORE is the packet state carried as metadata. Although C-SCORE does not maintain per-flow state at core nodes, it operates as a fair queuing scheduler, a class of schedulers known to provide the best flow isolation. The metadata carried in the packet header is simple and can be updated either while the packet waits in the queue or before it enters the queue. To guarantee its performance, C-SCORE, like any other DetNet solution, requires that the total service rate of the flows on each link remain below the capacity of that link. To ensure this, admission control is typically performed for each flow before data transmission. Admission control consists of the following steps:

signaling the service rate of the incoming flow;
checking the current total service rate of each link along the path; and
deciding whether to accept the flow.

specifies a framework, based on Y.3129, for identifying the service rates of active flows for admission control without maintaining per-flow state. This stateless service rate estimation improves the scalability of the admission control process. The framework relies on the metadata specified in Y.3129. [Editor's Note: The current versions of Y.3129 and Y.3148 define three metadata fields. They are expected to be revised to define four.] This document specifies the implementation details for realizing C-SCORE, including:

the metadata and its encoding formats;
the admission control procedures; and
an approximation of stateless fair queuing using a strict priority (SP) scheduler.

BE: Best Effort C-SCORE: Work Conserving Stateless Core Fair Queuing DetNet: Deterministic Networking E2E: End to End FQ: Fair Queuing FT: Finish Time GPS: Generalized Processor Sharing HoQ: Head of queue FIFO: First-In First-Out MNA: MPLS Network Actions NAS: Network Action Sub-Stack PIFO: Push-In First-Out PRPS: Packetized Rate Proportional Servers PSD: Post-Stack Data RSpec: Requested Specifications TLV: Type-Length-Value TSpec: Traffic Specifications VC: Virtual Clock

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Assumptions This document assumes that there are only two classes of traffic. High-priority traffic, equivalently DetNet traffic, requires guaranteed latency upper bounds. All other traffic is considered low-priority, or Best Effort (BE), traffic. This document addresses only high-priority traffic. However, a BE packet in transmission is not preempted by high-priority traffic. All flows of DetNet traffic conform to their traffic specification (TSpec) parameters. That is, for flow i with maximum burst size B_i and arrival rate a_i, the cumulative arrival from flow i during any time interval [t_1, t_2], where t_1 < t_2, does not exceed B_i + (t_2 - t_1)a_i. The service rate actually allocated to flow i, denoted r_i, is greater than or equal to its arrival rate a_i. As shown in Equation (4) in , the E2E latency bound of a flow can be adjusted by adjusting its service rate. The symbol r is used interchangeably with r_i to denote the service rate of the flow under observation. Similarly, B, a, and L are used to denote the parameters of the flow under observation. The total service rate allocated to all flows at a node does not exceed the capacity of the node's output link. Consequently, resource reservation and admission control are required. The following terms are used in this document:

Node: An output port module of a switching device. The term "server" is used equivalently.
Entrance node: For a given flow, the node at the edge of the network through which the flow enters the network.
Core node: For a given flow, a node in the network that the flow traverses and that is not the entrance node of that flow. A single node can be the entrance node for one flow and a core node for another flow.
Arrival and service of a packet: A packet is considered to have arrived at a node when its last bit is received by the node, and to have been serviced when its last bit is transmitted from the node.

Framework

Core Stateless Fair Queuing Fair queuing (FQ) schedulers assign each packet a Finish Time (FT), which determines its service order: the packet with the smallest FT in the buffer is served first. The key idea of FQ is to compute the ideal service completion time of each packet in a hypothetical fluid service model and to use it as the service order in the actual packet-based scheduler. Although FQ schedulers provide excellent flow isolation, they require per-flow state, namely F(p-1), the FT of the preceding packet of the same flow. Upon the arrival of each packet, the scheduler must identify the flow and retrieve the FT of the preceding packet of that flow. The scheduler must also update the flow state to F(p) upon the departure of the packet. and specify an FQ scheduler that does not require per-flow state at core nodes. At the entrance node, the FT of packet p is computed as shown in Equation (1), where the subscript 0 denotes the entrance node of the flow under observation. where:

A_0(p): The arrival time of packet p at the entrance node.
L(p): The length of packet p.
r: The service rate allocated to the flow.

The FT of packet p for core node h is obtained by adding a delay factor d_(h-1)(p), which depends on the upstream node h-1 and on the packet, to the FT for node h-1, as shown in Equation (2). The framework achieves a bounded E2E latency by defining the delay factor as shown in Equation (3). where:

L_h: The maximum packet length observed at node h over all flows.
R_h: The link capacity of node h.
L: The maximum packet length of the flow.
r: The service rate allocated to the flow.
delta_h(p): The time difference of packet p between node h and node h+1, defined as the arrival time of p at node h+1 minus the service completion time of p at node h (see ).

The time difference delta_h(p) includes the clock discrepancy and the propagation delay between node h and node h+1. Since it is relatively stable across packets, it can be approximated by a constant, delta_h. When d_h(p) is defined as in Equation (3), the latency bound in Equation (4) holds . where D_h(p) is the latency experienced by packet p from its arrival at node 0 to its departure from node h, and B is the maximum burst size of the flow to which p belongs. The last term of Equation (4) accounts for the clock discrepancies and the propagation delays between the nodes. When the clock discrepancies are negligible, for example, with an absolute time reference, this term effectively represents the E2E propagation delay. The latency bound in Equation (4) is the same as that of a network in which every node employs a stateful FQ scheduler. All parameters in the bound are intrinsic to the flow, except L_h/R_h and the propagation delays.

Stateless Service Rate Estimation The C-SCORE framework can also be used to estimate the total service rate of the flows on a link. This is based on the following two observations:

The FTs of consecutive packets of a flow partition the time axis into non-overlapping intervals. Therefore, for any given time instant, exactly one packet of the flow has an interval that contains that instant.
The difference between the FTs of consecutive packets of a flow remains the same across nodes, since, with delta_h approximated as a constant, the FTs of all packets of the flow are increased by the same amount at each node.

If each packet carries both its FT and the difference from the FT of the preceding packet, the packet whose interval contains a given time instant can be identified from the packet metadata alone. By choosing a reference time for estimation, exactly one such packet is identified per flow. Since the service rate of the flow is also carried in the packet as metadata, the service rates of these packets can be summed to obtain the total service rate of the link. specifies such a framework, based on , for identifying the service rates of active flows for admission control without maintaining per-flow state. This framework relies on the metadata specified in , in particular the difference between the FTs of the preceding and current packets, denoted as DeltaF(p) in this document.

Metadata , , and specify the following metadata to be carried in each packet of a flow:

L: The maximum packet length of the flow.
r: The service rate allocated to the flow.
F_h(p): The FT of packet p to be used at node h.
DeltaF(p): The difference between the FT of packet p and the FT of the preceding packet of the same flow, that is, F_0(p) - F_0(p-1).

L, r, and DeltaF(p) are set by the entrance node and remain unchanged along the path. F_h(p) is dynamic and is updated at every node in accordance with Equation (2). [Editor's Note: The current versions of Y.3129 and Y.3148 define three metadata fields: L, r, and F_h(p). They are expected to be revised to also include DeltaF(p).] When packet p arrives at core node h, it carries F_h(p), L, r, and DeltaF(p). F_h(p) has been calculated in advance by node h-1 using d_(h-1)(p), as shown in Equation (2). Node h uses F_h(p) to determine the service order of the packet. Before transmitting the packet, node h calculates F_(h+1)(p) = F_h(p) + d_h(p) and replaces F_h(p) with F_(h+1)(p) in the packet. As shown in Equation (3), d_h(p) is the sum of the flow-specific term L/r, which is obtained from the packet metadata, and the node-specific terms L_h/R_h and delta_h, which are maintained by node h. This update can be performed either before the packet enters the queue or while the packet waits in the queue. The entrance node calculates F_0(p) using Equation (1) and uses it to determine the service order of the packet at the entrance node. It then calculates F_1(p) = F_0(p) + d_0(p) and records F_1(p), together with L, r, and DeltaF(p), in the packet. The detailed operations for creating and updating the metadata at the entrance node and core nodes are specified in and . These references also describe considerations for reducing the complexity of the entrance node.

Header Format

IPv6 Header Format The IPv6 Hop-by-Hop (HbH) Options header should be used for carrying the metadata. It is a specific type of Extension Header designed for information that must be examined and processed by all the transit nodes along a packet's delivery path. This header is identified by a Next Header value of 0 in the IPv6 main header. The HbH Options header consists of a sequence of variable-length options, which are encoded in a Type-Length-Value (TLV) format. In this document, the metadata (L, r, F_h(p), and DeltaF(p)) are stored in the Option Data fields of four separate options within a single HbH Options header, as shown in . The Option Type and Opt Data Len fields specify the C-SCORE metadata identifier and the length of the metadata, respectively. The Option Type values are to be determined. Since F_h(p) is updated at every hop, the Option Type for F_h(p) should have the third-highest-order bit set to 1, indicating that the Option Data may change en route . A PadN option is added as necessary so that the length of the HbH Options header is a multiple of 8 octets. In the example in , the HbH Options header is 32 octets long, including a 4-octet PadN option, and the Hdr Ext Len field is set to 3.

IPv6 HbH Options Header Example for Four C-SCORE Metadata The following is the operational procedure in a transit node. The hardware parser sees Next Header 0 in the main IPv6 header and directs the packet to the metadata management function. This functional entity scans for the Option Types corresponding to C-SCORE. L (16 bits, TBD), r (32 bits, TBD), F_h(p) (48 bits, TBD), and DeltaF(p) (48 bits, TBD) are extracted and processed, and the new FT is calculated. Since the offset of the FT field is fixed relative to the start of the HbH Options header, the functional entity can update the field with a single-cycle write before forwarding the packet. No checksum recalculation is required, since IPv6 has no header checksum and extension headers are not covered by upper-layer checksums. Note that L, the maximum packet length of the flow, does not need to be precise at the byte level. Similarly, r, the service rate of the flow, can be of a coarse granularity or even several discrete predefined levels. [Editor's Note: The encoding of these metadata, especially their lengths and Option Type values, will be specified in a future version of this document.]

MPLS Label Format specifies formats and mechanisms for using MPLS Network Actions (MNA) to support DetNet services, including bounded latency, low loss, and in-order delivery. It specifies three information elements of DetNet packets: Flow identifier (Flow-ID), Sequence information (SeqNum), and Latency information (LatencyInfo). The C-SCORE metadata correspond to the Latency information. Two approaches for carrying information elements are specified: In-Stack and Post-Stack MNA. With In-Stack MNA, the DetNet-specific information is embedded directly within the MPLS label stack, as part of a Network Action Sub-Stack (NAS), before the Bottom of Stack (BoS). A Network Action Indicator (NAI) signals that the subsequent Label Stack Entries (LSEs) in the sub-stack carry ancillary data rather than forwarding labels. With Post-Stack MNA, the DetNet-specific information is carried after the label stack, between the BoS and the start of the user payload, and an indicator in the label stack signals its presence. Post-Stack MNA is better suited for large or variable-sized data that would otherwise make the label stack excessively deep. This document follows the Post-Stack approach, but the In-Stack approach is not excluded. The Post-Stack MNA solution consists of two components:

The Post-Stack MPLS Header (PSMH) Presence Bit (P bit) carried in the In-Stack NAS. Bit 20 of LSE Format B in the NAS is defined as the P bit, which indicates the presence of the PSMH after the BoS.
The PSMH, which consists of the Post-Stack MPLS Base Header and one or more Post-Stack Network Actions (PSNAs). The Post-Stack MPLS Base Header contains the Post-Stack First Nibble (PFN), the PSMH-Len field, and the Post-Stack MPLS Header Type. PSMH-Len specifies the length of the PSMH in 4-octet units, excluding the 4-octet Post-Stack MPLS Base Header.

The C-SCORE metadata, L, r, F_h(p), and DeltaF(p), are carried in four separate PSNAs with the Hop-by-Hop scope, so that they are processed at every hop. The number of bits required for each metadata field is for further study. The Post-Stack Network Action Length (PS-NAL) of each PSNA is determined by the length of the metadata. For example, if FT is 48 bits long, the PSNA for FT has a PS-NAL of 1, as shown in .

Post-Stack MNA Example with Four PSNAs for Four C-SCORE Metadata In the example in , the PSMH is located immediately after the BoS, that is, at a start offset of 0. The fields are set as follows:

P bit = 1: In the In-Stack NAS, indicates the presence of the PSMH.
PSMH-Len = 7: The four PSNAs occupy seven 4-octet units, excluding the Post-Stack MPLS Base Header.
MNA-PS-OP1 (TBD): The Post-Stack MNA Opcode for L. PS-NAL = 0, since the 16-bit L fits in the first 4 octets of the PSNA.
MNA-PS-OP2 (TBD): The Post-Stack MNA Opcode for r. PS-NAL = 1, since the 32-bit r requires one additional 4-octet unit, of which 16 bits are padding.
MNA-PS-OP3 (TBD): The Post-Stack MNA Opcode for F_h(p). PS-NAL = 1 for a 48-bit FT.
MNA-PS-OP4 (TBD): The Post-Stack MNA Opcode for DeltaF(p). PS-NAL = 1 for a 48-bit DeltaF(p).

The padding rules for Post-Stack Data that does not fill a multiple of 4-octet units are described in the document that defines the corresponding network action . For the C-SCORE metadata, the padding rules will be specified in a future version of or of this document. Note that, because of its stateless nature, C-SCORE does not require a Flow-ID to be carried in the packet.

Admission Control for Latency Guarantee As described in , the total service rate allocated to the flows on each link must not exceed the capacity of the link. This section describes recommended admission control procedures for the C-SCORE framework. The procedures use RSVP-like signaling with a PATH message sent from the entrance node toward the egress node and a RESV message sent in the reverse direction. This document does not mandate a specific signaling protocol or message encoding. In both procedures, only the entrance node maintains per-flow state. Core nodes do not maintain per-flow state; they obtain the total service rate of their outgoing links using the stateless service rate estimation mechanism specified in (see ).

Signaling Information A flow request from an application includes the following information:

Traffic Specification (T-Spec): The arrival rate a, the maximum packet length L, and the maximum burst size B of the flow.
Requested Specification (R-Spec): The E2E latency bound D_req and, optionally, the E2E jitter bound required by the application.

In addition to the T-Spec and R-Spec, a PATH message carries the following fields, which are updated hop by hop:

Hop count (H): The index of the last node that has processed the PATH message. The entrance node sets H to 0, and each subsequent node increments H by 1. When the PATH message reaches the egress node, the path consists of H+1 nodes, indexed from 0 to H.
Accumulated link term (S_LR): The sum of L_h/R_h over the nodes that have processed the PATH message.
Accumulated time difference (S_delta): The sum of delta_h over the links that the PATH message has traversed. Each node obtains delta_h as described in .
Requested service rate (r): Used in the basic procedure ().
Desired Rate (r_desired), Minimum Acceptable Rate (r_min), and Path-Available-Rate (r_path): Used in the rate discovery procedure ().

Using these fields, the E2E latency bound in Equation (4) for a service rate r can be written as follows: Accumulating S_LR and S_delta separately, rather than a single per-hop latency value, allows the egress node to evaluate the latency bound for any candidate service rate. This is required for the rate discovery procedure.

Basic Admission Procedure In the basic procedure, the application requests a specific service rate r, where r >= a.

The application requests a flow from the entrance node with the T-Spec, the R-Spec, and the requested service rate r.
The entrance node acts as the admission controller. It checks whether r is within the available capacity of its own outgoing link. If so, it creates tentative per-flow state for the flow, sets H = 0, S_LR = L_0/R_0, and S_delta = 0, and sends a PATH message toward the egress node. Otherwise, it rejects the request.
Each core node h estimates the total service rate of its outgoing link using the mechanism specified in and checks whether adding r keeps the total within the aggregate bandwidth threshold of the link. If not, the core node returns a PATH error message to the entrance node and does not forward the PATH message.
If the check succeeds, the core node increments H, adds delta_(h-1) to S_delta and L_h/R_h to S_LR, and forwards the PATH message to the next node.
The egress node performs the same check and update as a core node, and then computes the E2E latency bound D(r) using Equation (5). If D(r) <= D_req, the egress node generates a RESV message carrying r. Otherwise, it returns a PATH error message to the entrance node.
The egress node sends the RESV message to the entrance node. Since core nodes do not maintain per-flow state, they do not need to process the RESV message.
When the RESV message reaches the entrance node, the flow is admitted. The entrance node commits r to its per-flow state and starts marking the packets of the flow with the metadata specified in this document, performing initial shaping if necessary.

Admission Procedure with Rate Discovery The RSVP-like signaling can also be used to discover and negotiate an appropriate service rate for a flow. The PATH message identifies the link with the smallest residual capacity on the path, so the maximum rate that can be supported without violating the latency bounds of existing flows is determined in a single round trip.

The application requests a flow from the entrance node with the T-Spec, the R-Spec, a Desired Rate r_desired, and a Minimum Acceptable Rate r_min, where r_desired >= r_min >= a.
The entrance node acts as the admission controller. It computes the residual capacity r_avail_0 of its own outgoing link, sets r_path = min(r_desired, r_avail_0), and initializes H, S_LR, and S_delta as in . If r_path < r_min, it rejects the request. Otherwise, it creates tentative per-flow state and sends a PATH message toward the egress node.
Each core node h computes the residual capacity r_avail_h of its outgoing link as the aggregate bandwidth threshold of the link minus the total service rate of the flows already admitted. The total service rate is estimated using the mechanism specified in .
The core node updates r_path = min(r_path, r_avail_h). If the updated r_path < r_min, the core node returns a PATH error message to the entrance node and does not forward the PATH message. Otherwise, it updates H, S_LR, and S_delta as in and forwards the PATH message.
When the PATH message reaches the egress node, r_path represents the maximum service rate that the entire E2E path can support at that moment.
The egress node selects the service rate r_opt as described in . If no rate satisfies the conditions, it returns a PATH error message to the entrance node.
The egress node sends a RESV message carrying r_opt to the entrance node.
When the RESV message reaches the entrance node, the flow is admitted.
The entrance node commits r = r_opt to its per-flow state and starts marking the packets of the flow.

Selection of the Service Rate The highest supportable rate is not always the most appropriate rate. The egress node selects r_opt such that r_opt is not greater than r_path and not smaller than the arrival rate a, the Minimum Acceptable Rate r_min, and the rate required for the E2E latency bound in Equation (5) to satisfy D_req. If no such rate exists, the egress node rejects the request. Within this range, the egress node can select r_opt based on the following criteria:

Latency requirements: A higher service rate results in smaller FT increments and thus a smaller per-hop latency bound. A rate higher than the minimum required rate provides a margin against variations in delta_h.
Buffer constraints: The buffer required at core nodes depends on the maximum burst size B and the service rate r. The selected rate needs to be consistent with B so that the buffers at core nodes do not overflow.
Network availability: The egress node can choose a rate lower than r_path to leave capacity for flows that may join later.

Since core nodes need only the committed rate r carried in the packet metadata, the rate discovery and selection logic is confined to the control plane and the edge nodes. In large-scale networks, the same mechanism can be used for aggregate rate discovery, where r_path represents the available capacity for a bundle of flows that share the same path. In this way, the admission process maps the current congestion state of the path onto a single service rate parameter.

Operational Considerations

Concurrent admission requests: The service rate estimation in reflects a flow only after its packets are transmitted. If multiple PATH messages traverse the same link at about the same time, each of them may observe the same residual capacity, and the link may be overbooked. To prevent this, a core node can maintain a single aggregate counter of pending reservations, rather than per-flow state. The core node adds the requested rate (r, or r_path in the rate discovery procedure) to the counter when it forwards a PATH message and includes the counter when computing the residual capacity. The rate is removed from the counter after a timeout that is long enough for the packets of the newly admitted flow to be reflected in the service rate estimation.
Error handling: When the entrance node receives a PATH error or RESV error message, it removes the tentative per-flow state and notifies the application that the request was rejected. The application can retry with relaxed requirements, for example, a larger D_req or a smaller r_min.
Flow removal: When a flow terminates, the entrance node removes the per-flow state and stops marking the packets of the flow. Since core nodes do not maintain per-flow state, no explicit teardown message is required at core nodes. The flow is no longer counted by the service rate estimation once its last packet has been transmitted.
Idle flows: An admitted flow that is temporarily idle may not be counted by the service rate estimation. If the flow resumes transmission, the total service rate of a link may exceed the aggregate bandwidth threshold. Operators can mitigate this by setting the aggregate bandwidth threshold sufficiently below the link capacity, or by having the entrance node maintain the presence of the flow during idle periods.

Compensation of Time Difference between Nodes As described in , there is a time difference between adjacent nodes, which includes the clock discrepancy and the propagation delay between them. The time difference of packet p between node h and node h+1 is defined as follows: where A_(h+1)(p) is the arrival time of packet p measured by the clock of node h+1, and C_h(p) is the service completion time of packet p measured by the clock of node h. The FT does not need to be precise, since it is used only to determine the service order of packets. If the propagation delay between the nodes is constant and the clocks of the nodes do not drift relative to each other, delta_h(p) is the same for all packets and can be replaced by a constant, delta_h. In practice, clocks may drift slowly. Therefore, delta_h needs to be updated only occasionally, for example, periodically or when a change in the measured value exceeds a configured threshold.

Measurement of Time Difference To obtain delta_(h-1), node h needs C_(h-1)(p), the service completion time of a packet at the upstream node h-1. Node h computes delta_(h-1) from C_(h-1)(p) and its own measurement of A_h(p), using Equation (6). Since both the clock discrepancy and the propagation delay are included in delta_(h-1), node h does not need to separate them. C_(h-1)(p) can be provided to node h in one of the following ways:

Piggybacking: Node h-1 records C_(h-1)(p) in the metadata of selected packets. C_(h-1)(p) can be provided on demand, at the request of node h, or periodically.
Explicit messages: Nodes h-1 and h exchange dedicated timing messages. The message format can follow that of the Network Time Protocol (NTP) . However, the procedure is simpler than that of NTP. NTP estimates the clock offset alone, using round-trip measurements under the assumption of symmetric delay. Here, only the combined one-way value delta_(h-1) is required, so a single one-way measurement is sufficient.

Time Difference between Non-adjacent Nodes The time difference between non-adjacent nodes can be obtained in the same manner. This is useful when nodes that do not support C-SCORE exist between two C-SCORE nodes. In this case, however, the measured time difference also includes the variable queuing delays at the non-compliant nodes and is therefore not stable across packets. One possible approach is to sample the time difference over a sufficiently long interval and to use the maximum sampled value as delta_h. The latency bound in Equation (4) then holds only if the maximum queuing delay at the non-compliant nodes is bounded and is captured by the sampled maximum.

Characteristics

Taxonomy According to the taxonomy suggested by , the framework in this document, C-SCORE, is a flow-level, rate-based, work-conserving, asynchronous, non-periodic, and in-time solution. also defines seven suitable categories for deterministic networking. A category is defined as a set of solutions grouped by one or more criteria, where a criterion is a principle or standard by which a solution can be judged or assigned to a certain category. According to this categorization, C-SCORE belongs to the "flow-level rate-based unbounded" category, which is one of the seven suitable categories.

Strengths The dominant factor in the per-hop latency of C-SCORE is the maximum packet length of the flow divided by its service rate. This factor is independent of the parameters of other flows. As such, the most distinguishing strength of C-SCORE is its flow isolation capability. A fine-tuned E2E latency bound can be assigned to a flow by controlling the flow's own parameters, such as its service rate. Once a latency bound is assigned to a flow, it remains almost the same in spite of changes in the network situation, such as other flows joining and leaving. C-SCORE is work-conserving and thus benefits from statistical multiplexing gain without wasting bandwidth, which is a key to the success of the Internet. The consequence is a smaller average latency. The observed maximum latency is also much smaller than the theoretical latency bound. Note that, with a work-conserving solution, observing the theoretical latency bound is extremely difficult in real situations. This is because the worst-case latency results from a combination of multiple rare events, e.g., a maximum burst from a flow colliding with the maximum bursts from all other flows at every node. In contrast, with non-work-conserving solutions, latencies close to the bounds are commonly observed. C-SCORE is rate-based, and thus its admission control is simple, since the admission condition depends only on the service rates of flows. This process aligns well with existing protocols. Overall, C-SCORE is suitable for large-scale networks at any utilization level, with various types of flows joining and leaving dynamically.

Approximate C-SCORE Using Strict Priority Schedulers

General Description C-SCORE requires a priority queue that sorts the packets in the queue in accordance with their FTs. Compared with the strict priority (SP) schedulers used in current switching nodes, this may limit the overall data plane scalability as well as the deployability of the solution. SP schedulers are commonly implemented in the hardware chips of current switching nodes. They usually consist of 8 to 32 queues and serve packets in a higher-priority queue first whenever such packets are present. Packets in the same queue are served on a first-in, first-out (FIFO) basis. It is therefore desirable to implement C-SCORE with such an SP scheduler. This section specifies the architecture and algorithms of the approximate C-SCORE using rotating SP schedulers, as well as the E2E latency bound of a network of approximate C-SCORE nodes.

Transit Node Architecture This section specifies the architecture of an approximate C-SCORE transit node, in which an SP scheduler with a limited number of queues behaves as an approximate priority queue. The input port module classifies packets into DetNet flows and best-effort (BE) flows. Packets of DetNet flows are put into one of N queues in the output port module, which operate as rotating SP scheduler queues. If queue k, where 0 <= k <= N-1, has the highest priority at time t, then queue (k+N-1) mod N has the lowest priority at t. Packets of BE flows are put into a separate BE queue in the output port module. The BE queue is served only when there is no packet in queues 0 to N-1. Time is partitioned into discrete intervals of uniform length S, called slots. Each slot is allocated to one of the N queues. Let T_i denote the end boundary of slot i. An arriving packet p is assigned to slot i if its FT, F_h(p), falls within the interval (T_(i-1), T_i]. The packet is buffered in the corresponding queue and served in FIFO order within the queue. The SP scheduler gives precedence to the queue that corresponds to the earliest slot. At the end of each slot, the priorities of the queues rotate: the queue of the slot that has just ended becomes the lowest-priority queue and is reused for the slot N slots later. The scheduler is work-conserving and serves packets at the line rate whenever the system is backlogged. This section assumes that the FT of every packet falls within the range covered by the N slots. In other words, it is assumed that no packet has an FT beyond the slot boundaries covered by the N queues, or equivalently, that N*S is large enough to cover the FTs of all packets that can be present in the node at the same time.

Algorithms To provide differentiated quality of service, the per-hop latency needs to be adjusted according to the service rates of the flows. With a sufficiently small slot length S, the system can provide tiered latency bounds based on the number of slots spanned by L/r, the maximum virtual service interval of a flow. For a flow with (n-1)S < L/r <= nS, the per-hop latency in an idealized preemptive system is bounded by (n+1)S. Flows with higher service rates, that is, smaller L/r, therefore receive lower latency, which aligns latency performance with bandwidth allocation. In an idealized preemptive system, the scheduler ensures that all packets mapped to a slot are fully served before the end boundary of the slot, T_i. Consequently, the service interval of any packet is contained within its assigned slot. In a realistic non-preemptive system, however, the service completion can be delayed by up to L_h/R_h, where L_h is the maximum packet length and R_h is the link capacity at node h. Thus, the service of the packet is guaranteed to complete no later than T_i + L_h/R_h. The slot length S is chosen such that S >= min_p{L(p)/r(p)}. While a smaller S generally reduces the E2E latency, reducing S below this lower bound, which is determined by the minimum packet transmission time relative to the flow rate, yields no additional scheduling benefit. In practice, N hardware queues are allocated and managed cyclically to accommodate the slots. Consequently, the selection of S involves a trade-off: a larger S allows a smaller number of physical queues N, at the expense of reduced scheduling precision. The approximate C-SCORE follows the architecture of C-SCORE. That is, Equations (1) and (2) are still used. However, the delay factor in Equation (3) is replaced by the one in Equation (7). where S_h is the slot length at node h, and n_h is the integer that satisfies (n_h-1)S_h < L/r <= n_h*S_h for the flow under observation, that is, n_h = ceiling(L/(r*S_h)).

E2E Latency Bound of the Approximate C-SCORE In a network of approximate C-SCORE nodes, the E2E latency of packet p is bounded as shown in Equation (8). where D_H(p) is the latency experienced by packet p from its arrival at node 0 to its departure from node H, the last node on the path, and B is the maximum burst size of the flow. As in Equation (4), the last term accounts for the clock discrepancies and the propagation delays between the nodes. When the clock discrepancies are negligible, for example, with an absolute time reference, this term effectively represents the E2E propagation delay. Since n_j*S_j < L/r + S_j, the per-hop term (n_j+1)S_j in Equation (8) exceeds the corresponding term L/r in Equation (4) by less than 2S_j. Therefore, as S_j decreases, the latency bound of the approximate C-SCORE approaches that of C-SCORE, except for an additional L/r in the first term.

Considerations for Non-compliant Nodes A network can contain non-compliant end nodes and transit nodes. For example, some end nodes may not support the signaling required for admission control. The network can also contain legacy nodes, or nodes that implement data plane enhancement solutions other than C-SCORE. The compensation of the time difference across non-compliant transit nodes is described in . [Editor's Note: How the presence of non-compliant nodes can be compensated for, and how much these nodes degrade the performance of C-SCORE, will be discussed in a future version of this document.]

Relationships to ITU-T Standards This document is based on two ITU-T Recommendations, and , and one draft ITU-T Recommendation, . Recommendation ITU-T Y.3129 specifies the requirements and framework for C-SCORE. The requirements address the generation, update, and properties of the FT. The framework describes a mechanism that guarantees E2E latency bounds while meeting these requirements. It specifies how to obtain the FT at core nodes, how to select the delay factor, and how to configure a network for latency guarantees. Recommendation ITU-T Y.3148 specifies the functional architecture, functional entities, operational procedures, and packet metadata required to fulfill the requirements and framework specified in Recommendation ITU-T Y.3129. The operational procedures cover stateless FQ at the entrance node and core nodes, including the creation and update of the FT values of packets. Draft Recommendation ITU-T Y.qos-sre-rf specifies the requirements and framework for identifying the service rates of active flows for admission control without maintaining per-flow state, based on the packet metadata defined in . This document bridges the gap between these ITU-T standards and deployable protocols. Specifically, it specifies:

the metadata () and the corresponding header formats for IPv6 and MPLS ();
the admission control procedures (); and
the approximation of C-SCORE using SP schedulers ().

There might be matters that require IANA considerations associated with metadata. If necessary, relevant text will be added in a later version.

This section will be described later.