]> Futurewei Technologies USA

2220 Central Expressway Santa Clara CA 95050 USA tte@cs.fau.de

Huawei Technologies

Nanjing China liyizhou@huawei.com

University of Surrey ICS

United Kingdom s.bryant@surrey.ac.uk

Malis Consulting

USA agmalis@gmail.com

ETRI

South Korea ryoo@etri.re.kr

China Mobile

China liupengyjy@chinamobile.com

Huawei Technologies

Beijing China liguangpeng@huawei.com

Huawei Technologies

Beijing China renshoushou@huawei.com

DETNET This memo specifies a forwarding method for bounded latency and bounded jitter for Deterministic Networks and is a variant of the IEEE TSN Cyclic Queuing and Forwarding (CQF) method. Tagged CQF (TCQF) supports more than two cycles and indicates the cycle number in an existing or new packet header field, called the tag. The tag replaces the cycle mapping of CQF, which is based purely on the synchronized reception clock. This memo standardizes TCQF as a mechanism independent of the tagging method used. It also specifies tagging via (1) the existing MPLS packet Traffic Class (TC) field for MPLS packets, (2) the IP/IPv6 DSCP field for IP/IPv6 packets, and (3) a new TCQF Option header for IPv6 packets. Target benefits of TCQF include low end-to-end jitter, ease of high-speed hardware implementation, the optional ability to support a large number of flows in large networks through DiffServ-style aggregation by applying TCQF to the DetNet aggregate instead of each DetNet flow individually, and support of wide-area DetNet networks with arbitrary link latencies and latency variations as well as low accuracy clock synchronization.

Introduction

Terminology

CQF

Cyclic Queuing and Forwarding. A queuing mechanism defined by annex T of .

DT

Dead Time. A term from CQF indicating the time during each cycle in which no frames can be sent because the receiving node could not receive them into the desired cycle buffer.

TCQF

Tagged Cyclic Queuing and Forwarding. The mechanism specified in this memo.

Summary of TCQF benefits and goals for DetNet (informative) For background information on CQF, its benefits and challenges for DetNet, and how multiple buffers and tagging address them, see . TCQF inherits the benefits of CQF for DetNet as outlined in , and by using a configurable number of three or more cycles, and signaling the cycle as part of a packet header, it resolves these problems as follows. With three cycles, TCQF can support arbitrary latency links at arbitrary speeds without reduction of utilization because of longer links or higher link speeds (same cycle time, same clock accuracy, only change in lengths and speeds). With four or more cycles, TCQF can also eliminate Dead Time caused by variation of clock synchronization inaccuracies (MTIE) as well as jitter caused by link propagation and processing variation. The sum of cycle times needs to be larger than the total jitter to achieve this. Prior documents describing the concept of TCQF (without using that name) include and . TCQF does not depend on other elements of , so it can also be used standalone in otherwise non-deterministic IP/IPv6 or MPLS networks to achieve bounded latency and low jitter. TCQF is especially beneficial when networks are architected to avoid per-hop, per-flow state even for traffic steering, which is the case for networks using SR-MPLS for traffic steering of MPLS unicast traffic, SRv6 for traffic steering of IPv6 unicast traffic and/or BIER-TE for tree engineering of MPLS multicast traffic by using the TC and/or DSCP header fields of BIER packets according to . In these networks, it is specifically undesirable to require per-flow signaling to non-edge forwarders (such as P-LSR in MPLS networks) solely for DetNet QoS because such per-flow state is unnecessary for traffic steering and would be required only for the bounded latency QoS mechanism, likely requiring even more complex hardware and manageability support than what was previously required for per-hop steering state (such as in RSVP-TE, ). Note that the DetNet architecture does not include full support for this DiffServ model, which is why this memo describes how to use TCQF with the DetNet architecture per-hop, per-flow processing as well as without it.

Using TCQF in the DetNet Architecture and MPLS forwarding plane (informative) This section gives an overview of how the operations of TCQF relate to the DetNet architecture. We first revisit QoS with DetNet in the absence of TCQF using an MPLS network as an example.

| Appl. | +----------+ +---------+ +---------+ +----------+ | Service |<--| Service |-- DetNet flow --| Service |-->| Service | +----------+ +---------+ +----------+ +---------+ +----------+ |Forwarding| |Fwd| |Fwd| |Forwarding| |Fwd| |Fwd| |Forwarding| +-------.--+ +-.-+ +-.-+ +----.---.-+ +-.-+ +-.-+ +---.------+ : Link : / ,-----. \ : Link : / ,-----. \ +........+ +-[ Sub- ]-+ +......+ +-[ Sub- ]-+ [Network] [Network] `-----' `-----' |<- LSP -->| |<-------- LSP -----------| |<--- LSP -->| |<----------------- DetNet MPLS --------------------->| ]]>

The above is copied from , Figure 2, and only enhanced by numbering the nodes to be able to better refer to them in the following text. Assume a DetNet flow is sent from T-PE1 to T-PE2 across S-PE1, LSR-P, S-PE2. In general, bounded latency QoS processing is then required on the outgoing interface of T-PE1 towards S-PE1, and any further outgoing interface along the path. When T-PE1 and S-PE2 know that their next-hop is a service LSR, their DetNet flow label stack may simply have the DetNet flow's Service Label (S-Label) as its Top of Stack (ToS) LSE, explicitly indicating one DetNet flow. On S-PE1, the next-hop LSR is not DetNet aware, which is why S-PE1 would need to send a label stack where the S-Label is followed by a Forwarding Label (F-Label), and LSR-P would need to perform bounded latency based QoS on that F-Label. For bounded latency QoS mechanisms relying on per-flow regulator state (aka: per-flow packet scheduling), such as in , this requires the use of per-DetNet-flow F-Labels across the network from S-PE1 to S-PE2. These could for example be assigned/managed through RSVP-TE enhanced as necessary with QoS parameters matching the underlying bounded latency mechanism (such as ). With TCQF, a sequence of LSR and DetNet service nodes implement TCQF with MPLS TC, ideally from T-PE1 (ingress) to T-PE2 (egress). The ingress node needs to perform per-DetNet-flow per-packet "shaping"/"regulating" to assign each packet of a flow to a particular TCQF cycle. This is specified in . All LSR/Service nodes after the ingress node only have to map a received TCQF tagged DetNet packet to the configured cycle on the output interface, not requiring any per-DetNet-flow QoS state. These LSR/Service nodes therefore also do not require per-flow interactions with the controller plane for the purpose of bounded latency. Per-flow state therefore is only required on nodes that are DetNet service nodes, or when explicit, per-DetNet flow steering state is desired, instead of ingress steering through e.g.: SR-MPLS. Operating TCQF per-flow stateless across a service node, such as S-PE1, S-PE2 in the picture is only one option. It is of course equally feasible to have one TCQF domain from T-PE1 to S-PE1, start a new TCQF domain there, running for example up to S-PE2, and start another one from there to T-PE2. A service node must act as an egress/ingress edge of a TCQF domain if it needs to perform operations that change the timing of packets other than the type of latency that can be considered in configuration of TCQF (see ). For example, if T-PE1 is ingress for a TCQF domain, and T-PE2 is the egress, S-PE1 could perform the DetNet Packet Replication Function (PRF) without having to be a TCQF edge node as long as it does not introduce latencies not included in the TCQF setup and the controller plane reserves resources for the multitude of flows created by the replication taking the allocation of resources in the TCQF cycles into account. Likewise, S-PE2 could perform the Packet Elimination Function (PEF) without being a TCQF edge node, as this most likely does not introduce latency that TCQF cannot accommodate, and the controller plane accordingly reserves resources for only one flow on the S-PE2->T-PE2 leg. If, on the other hand, S-PE2 were to perform the Packet Ordering Function (POF), this could create large peaks of packets when out-of-order packets are released together. A POF would either have to shape these bursts so that the traffic of the flow again conforms to the admitted CIR/PIR, or else the service node would have to be a TCQF egress/ingress, performing that shaping itself as an ingress function.

TCQF per-flow stateless forwarding (normative)

Configuration Data model and tag processing for MPLS TC tags The following data model summarizes the configuration parameters as required for TCQF and discussed in the following sections. 'tcqf' includes the parameters independent of the tagging on an interface. 'tcqf_*' describes the parameters for interfaces using MPLS TC and IP DSCP tagging.

Packet processing This section explains the TCQF packet processing and through it, introduces the semantics of the objects in . tcqf contains the router wide configuration of TCQF parameters, independent of the specific tagging mechanism on any interface. Any interface can have a different tagging method. This document uses the term router when it is irrelevant whether forwarding is for IP or MPLS packets, and the term Label Switched Router (LSR) to indicate MPLS is used, or IP router to indicate IP or IPv6 are used - independent of the specific encapsulation used for IP or MPLS to carry the cycle identification. The model represents a single TCQF domain, which is a set of interfaces acting both as ingress (iif) and egress (oif) interfaces, capable of forwarding TCQF packets between each other. A router may have multiple TCQF domains each with a set of interfaces disjoint from those of any other TCQF domain. tcqf.cycles is the number of cycles used across all interfaces in the TCQF domain. Routers MUST support 3 and 4 cycles. The maximum number of supportable cycles depends on the encapsulation. For example, to support interfaces with MPLS TC tagging, all interfaces in the TCQF domain MUST NOT use more than 7 cycles. See . The unit of tcqf.cycle_time is micro-seconds. Routers MUST support configuration of cycle-times of 20,50,100,200,500,1000,2000 usec. Cycles start at an offset of tcqf.cycle_clock_offset in units of nsec as follows. Let clock1 be a timestamp of the local reference clock for TCQF, at which cycle 1 starts, then: tcqf.cycle_clock_offset = (clock1 mod (tcqf.cycle_time * tcqf.cycles) ) The local reference clock of the router is expected to be synchronized with the neighboring LSR/router in TCQF domain. tcqf.cycle_clock_offset may be configurable by the operator, or it may be read-only. In either case, the operator is able to configure working TCQF forwarding through appropriately calculated cycle mappings. tcqf.if_config[oif] is an optional per-interface configuration of TCQF parameters. tcqf.if_config[oif].cycle_clock_offset may be different from tcqf.cycle_clock_offset, for example, when interfaces are on line cards with independently synchronized clocks, or when non-uniform ingress-to-egress propagation latency over a complex router/LSR fabric makes it beneficial to allow per-egress interface or line card configuration of cycle_clock_offset. It may be configurable or read-only. The value of -1 for tcqf.if_config[oif].cycle_clock_offset is used to indicate that the domain wide tcqf.cycle_clock_offset is to be used for oif. This is the only permitted negative number for this parameter. When a packet is received from iif with a cycle value of iif_cycle and the packet is routed towards oif, then the cycle value (and buffer) to use on oif is tcqf.if_config[oif].cycle_map[iif].oif_cycle[iif_cycle]. This is called the cycle mapping and must be configurable. This cycle mapping always happens when the packet is received with a cycle tag on an interface in a TCQF domain and forwarded to another interface in the same TCQF domain. This encapsulation independent data model only defines how to map from a received packet's cycle to a sending interface cycle buffer and hence sent packet cycle. It does not specify how the cycle identifier is encoded in the received or sent packet. This is specified in the following sections. Therefore, this data model also does not determine whether interfaces use IP/IPv6, MPLS or any other encapsulation. This is determined by the configuration of the DetNet domain. A mixed use of MPLS and IP/IPv6 interfaces is possible with this data model, but at the time of writing this document not supported by DetNet.

TCQF for MPLS with TC tagging This section describes operation of TCQF for MPLS packets using the Traffic Class (TC) field of the MPLS label to carry the cycle-id. To support this encapsulation, the TCQF Data Model as defined in is expanded as follows.

tcqf_tc[oif].tc[oif_cycle] defines how to map from the internal cycle number oif_cycle to an MPLS TC value on interface oif. tcqf_tc[oif] MUST be configured, when oif uses MPLS. This oif_cycle <=> tc mapping is not only used to map from internal cycle number to MPLS TC tag when sending packets, but also to map from MPLS TC tag to the internal cycle number when receiving packets. In the terminology of , TCQF QoS as defined here is TC-Inferred-PSC LSP (E-LSP) behavior: Packets are determined to belong to the TCQF PSC solely based on the TC of the received packet. The internal cycle number SHOULD be assigned from the Top of Stack (ToS) MPLS label TC bits before any other label stack operations happen. On the egress side, the TC value of the ToS MPLS label SHOULD be assigned from the internal cycle number after any label stack processing. With this order of processing, TCQF can support forwarding of packets with any label stack operations such as label swap in the case of LDP or RSVP-TE created LSP, Penultimate Hop Popping (PHP), or no label changes from SID hop-by-hop forwarding and/or SID/label pop as in the case of SR-MPLS traffic steering.

TCQF for IP/IPv6 with DSCP tagging This section describes operation of TCQF for IP/IPv6 packets using the Differentiated Services Code Point (DSCP) field of IP/IPv6 packets to carry the cycle-id. To support this encapsulation, the TCQF Data Model as defined in is expanded as follows.

tcqf_dscp[oif].dscp[oif_cycle] defines how to map from the internal cycle number oif_cycle to an IP/IPv6 DSCP value on interface oif. tcqf_dscp[oif] MUST be configured, when oif uses DSCP tagging of IP/IPv6 packets for TCQF. This oif_cycle <=> dscp mapping is not only used to map from internal cycle number to the DSCP tag when sending packets, but also to map from IP/IPv6 DSCP to the internal cycle number when receiving packets. Because DetNet domains are currently assumed to be networks of a single administrative operator domain, this document does not ask for standardization of the DSCP to use with TCQF. Instead, deployments wanting to use TCQF with IP/IPv6 encapsulation and DSCP tagging need to assign DSCPs within their domain from the xxxx11 "EXP/LU" Codepoint space according to , Section 6. This allows up to 16 DSCP for intradomain use and hence up to 16 cycle identifiers.

TCQF for IPv6 with IPv6 Option tagging This section describes operation of TCQF for IPv6 packets without having to rely on DSCP by defining a new IPv6 option for DetNet. This option is to be placed in the IPv6 HbH (Hop-by-Hop) Options or DOH (Destination Option Header) header. To support this encapsulation, the TCQF Data Model as defined in is expanded as follows.

TCQF Option Format The TCQF Option helps the receiving port to identify in which time cycle interval the packet is sent from the upstream router. It can be used to determine the output port cycle buffer to enqueue the packet.

TCQF-Option fields: Option Type: 8-bit identifier of the type of option. Value TBD by IANA. If the processing IPv6 node does not recognize the Option Type it must discard the packet and return an ICMPv6 message (the highest-order 2 bits = 10). The Option Data of this option may change en route to the packet's final destination (the third- highest-order bit=1). Opt Data Len: 8-bit length of the option data. Flags: 8-bit field to indicate what TCQF Option information follows. The leftmost bit is called E-bit. When E-bit set to 1, there is a 64-bit extension in length after Cycle Id. Cycle Id: 8-bit field to indicate the time cycle ID at output port of the upstream node when the packet is sent out. This is the packet header field name for the data model ipv6oh[oif_cycle] element. 64-bit extension: This field contains values required for possible additional options, such as timestamp. This field exists only when E-bit in Flags field is set to one. [Editor's Note: Text will be modified or added as specific uses for this field are identified]

TCQF Option Processing A packet carrying the TCQF Option with Cycle Id does not change the fundamental cyclic queuing and forwarding behaviors of TCQF over the encapsulation independent forwarding behavior described above (). Compared to DSCP it does not introduce a limited number of cycle-ids, and avoids the operational concern of using multiple DSCPs for what is effectively a single per-hop forwarding behavior, a novel aspect that could otherwise cause issues, for example with diagnostics or other operational standards. It also allows easier extensions with other potentially beneficial DetNet features in the same Option header. As part of the packet processing of , the Cycle ID field of the option header is rewritten from tcqf.ipv6oh[oif_cycle], in the same way as DSCP would be rewritten from tcqf.dscp[oif_cycle].

Encapsulation of TCQF Option for Deterministic IP (DIP) data plane When used in IPv6 () networks, the TCQF Option can be placed in an HbH extension header or Destination Option Header (DOH).

shows the encapsulation of TCQF option in HbH extension header for deterministic IP (DIP) data plane. When every DetNet forwarding node along the path is provisioned to use TCQF as the queuing mechanism, this option should be placed here. If a router does not support this option, it discards the packet and returns an ICMP message. In some deployments the path selection is indicated using IPv6 routing header (RH) by specifying a set of nodes that must be traversed by the packet along its path to the destination. When such a source routing mechanism is used, TCQF Option is placed in DOH (Destination Option Header) as shown in for Deterministic IP data plane. Then the TCQF Option will be processed by the specified in-path routers.

(TBD: Should the TCQF Option be used with SRv6, and if so, how?)

TCQF Pseudocode (normative) The following pseudocode restates the forwarding behavior of algorithmically. It uses the objects of the TCQF configuration data model defined in .

= nextcyclestart); // wait until next cycle nextcyclestart += tcqf.cycle_time forall(iif) { forall(pak = tcqf_dequeue(oif.cycleq[cycle,iif]) { schedule to send pak on oif before nextcyclestart; // [4] } } cycle = (cycle + 1) mod tcqf.cycles + 1 } } ]]>

Processing of ingress TCQF packets is performed via ingres_flow_enqueue(pak) and ingress_flow_2_tcqf(oif,cycle) as explained in . Packets in a cycle buffer can be sent almost arbitrarily within the time period of the cycle. They also do not need to be sent as soon as possible, as long as all will be sent within that period. There is no need to send them in the order of their arrival except that packets from the same ingress flow that end up in the same cycle must not be reordered across any number of TCQF hops. The pseudocode describes this by using a queue oif.cycleq[cycle,iif] ([3]) for all packets from the same iif. The pseudocode describes the otherwise arbitrary scheduling of all packets within the cycle time via the statement shown in [4]. Ingress packets are passed from their ingress queues to the next cycle queue via [5]. Processing of egress TCQF packets is out-of-scope. It can be performed by any non-TCQF packet forwarding mechanism such as some strict priority queuing in step [2], and packets could accordingly be marked with a corresponding packet header traffic class indicator for such a traffic class in step [1].

TCQF Per-flow Ingress forwarding (normative) In this section, ingress flows are flows whose packets enter the router from a non-TCQF interface and need to be forwarded to an interface with TCQF. In the simplest case, these packets are sent by the source and the router is the first-hop router. In another case, the router's ingress interface connects to a hop where the previous router(s) performed a different bounded latency forwarding mechanism than TCQF.

Cycle scheduling considerations The model specified in this document is the simplest possible ingress scheduling model for passing packets from flows into cycles. For each flow, a maximum number of bits is specified on the ingress router that the flow is allowed to put into a cycle. The controller plane is responsible for ensuring that the sum of csize of all admitted ingress flows will fit into each cycle. It likewise has to ensure that the cycle sizes of all flows across some intermediate router hop will fit into each TCQF cycle. Note that intermediate hops may have higher-bandwidth links, so the same cycle length does not necessarily mean the same cycle size in bits. Controller plane operations will become difficult with this scheduling model when the sum of burst sizes of flows can exceed the cycle size on the ingress router. This is especially true when carrying a larger number of flows with a low guaranteed bit-rate but a large burst size. For example, flows that need to send a single packet with a period much longer than a single cycle. Without further coordination across many of these flows, simple admission control would need to assume that all these packets could arrive at the same time, creating a burst of multiple cycles. provides a more refined ingress scheduling mechanism which solves this performance limitation of the simple ingress forwarding model presented here. It can be implemented either as a replacement for the ingress forwarding described here, or by prepending its policing of packets to the one assumed here (from ).

Ingress Flows Configuration Data Model

The data model shown in expands the tcqf data model from . For every DetNet flow for which this router is the TCQF ingress, the controller plane has to specify a maximum number of bits called csize (cycle size) that are permitted to go into each individual cycle. Note that iflow[flowid].csize is not specific to the sending interface because it is a property of the DetNet flow.

Ingress Flows Pseudocode When a TCQF ingress packet is received, it first has to be enqueued into a per-flow queue. This is necessary because the permitted burst size for the flow may be larger than what can fit into a single cycle, or even into all the cycles used in the network.

ingres_flow_enqueue(pak) as shown in performs this enqueuing of the packet. Its position in the DetNet/TCQF forwarding code is shown in . police(pak): If the router is not only the TCQF ingress router, but also the first-hop router from the source, ingres_flow_enqueue(pak) will also be the place where policing of the flow's packets according to the Traffic Specification of the flow would happen - to ensure that packets violating the Traffic Specification will not be forwarded, or be forwarded with lower priority (e.g.: as best effort). This policing and resulting forwarding action is not specific to TCQF and therefore out of scope for this text. See , section 5.5.

ingress_flow_2_tcqf(oif, cycle) as shown in transfers ingress DetNet flow packets from their per-flow queue into the queue of the cycle that will be sent next. The position of ingress_flow_2_tcqf() in the DetNet/TCQF forwarding code is shown in .

Operational Considerations

High-Speed Implementation High-speed implementations with programmable forwarding planes of TCQF packet forwarding require Time-Gated Queues for the cycle queues, such as introduced by and also employed in CQF . Compared to CQF, the required accuracy of clock synchronization across the nodes is lower, as explained in below. High-speed forwarding for ingress packets as specified in above would require packets to first be passed into a per-flow queue and then be re-queued into a cycle queue. This is not ideal for high speed implementations. The pseudocode for ingres_flow_enqueue() and ingress_flow_2_tcqf(), like the rest of the pseudocode in this document is only meant to serve as the most compact and, hopefully, easiest to read specification of the desired externally observable behavior of TCQF - but not as a guidance for implementation, especially not for high-speed forwarding planes. High-speed forwarding could be implemented with single-enqueueing into cycle queues as follows: Let B[f] be the maximum amount of data that the router would need to buffer for ingress flow f at any point in time. This can be calculated from the flow's Traffic Specification. For example, when using the parameters of , section 5.5.

Maxcycles is the maximum number of cycles required so that packets from all ingress flows can be directly enqueued into maxcycles queues. The router would then cycle not across tcqf.cycles queues, but across maxcycles queues, while still cycling across tcqf.cycles cycle tags. The calculation of B[f], and as a result maxcycles, may be further refined (lowered) by additionally known constraints such as the bitrates of the ingress interface(s) and TCQF output interface(s).

Controller plane computation of cycle mappings The cycle mapping is computed by the controller plane by taking at minimum the link, interface serialization and node internal forwarding latencies as well as the cycle_clock_offsets into account.

In , consider that router R1 sends packets via C = 3 cycles with a cycle_clock offset of O1 towards Router R2. These packets arrive at R2 with a cycle_clock offset of O1' which includes through D all latencies incurred between releasing a packet on R1 from the cycle buffer until it can be put into a cycle buffer on R2: serialization delay on R1, link delay, non-CQF delays in R1 and R2, especially forwarding in R2, potentially across an internal fabric to the output interface with the sending cycle buffers.

shows a formula to calculate the cycle mapping between R1 and R2, using the first available cycle on R2. In the example of with CT = 1, (O1' - O2) =~ 1.8, A will be 0, resulting in map(1) = 1, map(2) = 2 and map(3) = 3. The offset "C" for the calculation of A is included so that a negative (O1 - O2) will still lead to a positive A. In general, D will be variable [Dmin...Dmax], for example because of differences in serialization latency between min and max size packets, variable link latency because of temperature based length variations, link-layer variability (radio links) or in-router processing variability. In addition, D also needs to account for the drift between the synchronized clocks for R1 and R2. This is called the Maximum Time Interval Error (MTIE). Let A(d) be A where O1' is calculated with D = d. To account for the variability of latency and clock synchronization, map(i) has to be calculated with A(Dmax), and the controller plane needs to ensure that A(Dmin)...A(Dmax) covers at most (C - 1) cycles. If it covers C cycles, then C and/or CT are chosen too small, and the controller plane needs to use larger numbers for either. This (C - 1) limitation results from the fact that there is only one buffer for each cycle, so a cycle cannot receive packets when it is sending packets. While this could be changed by using double buffers, this would create additional implementation complexity and not solve the limitation for all cases, because the number of cycles to cover [Dmin...Dmax] could also be (C + 1) or larger, in which case a tag of 1...C would not suffice.

Implementation, Deployment and Validation considerations (informative)

Link speed and bandwidth sharing TCQF hops along a path do not need to have the same bitrate; they just need to use the same cycle time. The controller plane then has to take the TCQF capacity of each hop into account when admitting flows based on their Traffic Specification and TCQF csize. TCQF does not require allocation of 100% of the link bitrate. When TCQF has to share a link with other traffic classes, queuing just has to be set up to ensure that all data of a TCQF cycle buffer can be sent within the TCQF cycle time. This can be achieved, for example, by making the TCQF cycle queues the highest priority queues and then limiting their capacity through admission control to leave time for other queues to be served as well.

Controller-plane considerations TCQF is applicable to both centralized as well as decentralized/distributed controller-plane models. From the perspective of the controller plane, if it is centralized, then it is logically very simple to perform admission control for any additional flow by checking that there is sufficient bandwidth for the amount of bits required for the flow on every cycle along the intended path. Likewise, path computation can be done to determine on which non-shortest path those resources are available. More efficient use of resources can be achieved by considering that flows with low bit rates would not need bits reserved in every cycle, but only in every N'th cycle. This requires different gates on ingress to admit packets from such flows than shown in this document and more complex admission control that attempts for example to interleave multiple flows across different sets of cycles to utilize all cycles as well as possible. This is the same complexity as in TSN technologies. Besides admission control and different ingress policing, such enhancements have no impact on the per-hop TCQF forwarding and can thus potentially be added incrementally. Decentralized or distributed controller planes including on-path, per-flow signaling, such as one using the mechanisms of RSVP-TE, is equally feasible with TCQF. In this case one of the potential benefits of TCQF is not leveraged, which is the complete removal of per-hop, per-flow awareness on each router. Nevertheless, such a controller plane only introduces this state maintenance into the control plane of each router; it does not change the TCQF forwarding plane, which retains its per-hop, per-flow stateless nature and the resulting performance/cost benefits.

Validation describes an accurate simulation-based validation of TCQF and provides further details on the mathematical models. summarizes the validation of a commercial 100Gbps router implementation of TCQF, deployed and measured in a research testbed spanning up to 2000km across China, operated by the China Environment for Network Innovations (CENI). The report also provides a reference to a more detailed version of the report. Note that both reports are in Chinese. TCQF is called DIP in these reports.

Security Considerations TBD.

IANA Considerations This document defines a new TCQF-Variant Option for the “Destination Options and Hop-by-Hop Options” under the “Internet Protocol Version 6 (IPv6) Parameters” registry with the suggested values in .

Acknowledgement Many thanks to David Black (DetNet tech advisor) for his review. Thanks to Lou Berger for his comments on controller-plane applicability. Thanks to Aijun Wang for his suggestions on the structure of the document, to Donald Eastlake for his review and editorial comments, to Janos Farkas for his comments on the name of the mechanism, and to Jinoo Joung for his feedback, all during or after the WG adoption call. Thanks to Xiaoliang Zheng and Fan Yang for their input to the TCQF option format. They were initially listed as co-authors for draft-yizhou-detnet-ipv6-options-for-cqf-variant, which was merged into this document.

Open Issues [RFC-editor: please remove] Renaming of the mechanism (feedback from Janos Farkas). TCQF stands for "Tagged CQF", but CQF as specified in already operates on VLAN-tagged frames, so "tagged" may not be sufficiently distinctive, and a more precise, functional name was suggested (e.g.: "Labelled CQF", LCQF). The authors are hesitant to change the name because of how long the current name has been in use. The authors will happily run a call in the WG if there is more desire for a name change.

Changelog [RFC-editor: please remove] Initial draft name: draft-eckert-detnet-mpls-tc-tcqf 00 Initial version 01 Added new co-author. Changed Data Model to "Configuration Data Model", and changed syntax from YANG tree to a non-YANG tree, removed empty section targeted for YANG model. Reason: the configuration parameters that we need to specify the forwarding behavior is only a subset of what likely would be a good YANG model, and any work to define such a YANG model not necessary to specify the algorithm would be scope creep for this specification. Better done in a separate YANG document. Example additional YANG aspects for such a document are how to map parameters to configuration/operational space, what additional operational/monitoring parameter to support and how to map the YANG objects required into various pre-existing YANG trees. Improved text in forwarding section, simplified sentences, used simplified configuration data model. 02 Refresh 03 Added ingress processing, and further implementation considerations. New draft name: draft-eckert-detnet-tcqf 00 Added text for DSCP based tagging of IP/IPv6 packets, therefore changing the original, MPLS-only centric scope of the document, necessitating a change in name and title. This was triggered by the observation of David Black at the IETF114 DetNet meeting that with DetNet domains being single administrative domains, it is not necessary to have standardized (cross administrative domain) DSCP for the tagging of IP/IP6 packets for TCQF. Instead it is sufficient to use EXP/LU DSCP code space and assignment of these is a local matter of a domain as is that of TC values when MPLS is used. Standardized DSCP in the other hand would have required likely work/oversight by TSVWG. In any case, the authors feel that with this insight, there is no need to constrain single-domain definition of TCQF to only MPLS, but instead both MPLS and IP/IPv6 tagging can be easily specified in this one draft. 01 Added new co-author. 02 Attempt to resolve issues from https://github.com/toerless/detnet/issues/1. Review from David Black, refine queueing/scheduling of pseudocode/explanation to highlight the non-sequential requirements. Comment from Lou Berger re. applicability of controller-plane resulting in new section about controller-plane. Reference to CENI chinese validation deployment. 03 Merged draft with draft-yizhou-detnet-ipv6-options-for-cqf-variant-02. Changed specification to be independent of encapsulation/forwarding plane and moved MPLS and IP/DSCP (from old TCQF draft) and IPv6 with extension header into separate seconds. Human translation of CENI report, uploaded CENI report with permission from CENI onto web page accessible from outside chinese firewall. 04 Added appendix sections on comparison with CSQF and multi class TCQF 05 Refresh. 06 Refresh. 07 Refresh - still waiting for more discuss of direction in detnet WG. 08 Refresh - still waiting for more discuss of direction in detnet WG. 09 Added section "Solution Scope, Taxonomy and Status for DetNet adoption considerations", made normative and claim it to be part of the overall solution. 10, 11 Moved Xiaoliang Zheng and Fan Yang into acknowledgement section. Explanation: They where initially listed as co-authors for draft-yizhou-detnet-ipv6-options-for-cqf-variant which was merged into draft-eckert-detnet-tcqf and at that point in time the list of authors was simply merged. As part of preparing for possible adoption of draft-eckert-detnet-tcqf we revisited the amount of participation of the listed authors and concluded, that the input from Xiaoliang Zheng and Fan Yang was best met by giving them acknowledgment also noting that they have not been active in the IETF DetNet work itself. IETF adopted draft name: draft-ietf-detnet-tcqf 00 Initial version 01 Just refresh. Ran out of time incorporating outstanding feedback before IETF126. Sorry. Will get to it within a month. 02 Added section "Cycle scheduling considerations" to the per-flow ingress forwarding section: explains the simple per-flow csize ingress scheduling model, the resulting controller-plane responsibilities (sum of csize per cycle on ingress and on intermediate hops with possibly different link speeds), its limitation for large numbers of low-rate flows with large bursts, and how can replace or complement it. English editorial pass across the document, done with the help of claude.ai (no intended technical changes). Includes: terminology fix Packet Reordering Function (PRF) -> Packet Ordering Function (POF) and added Packet Elimination Function (PEF) abbreviation; fixed TCQF domain example (T-PE1 to S-PE1); reworded MPLS TC cycle limit sentence ("MUST NOT use more than 7 cycles") per review from Donald Eastlake; fixed duplicate list numbering, spelling and punctuation. Restructured the document following suggestions by Aijun Wang during the WG adoption call: moved former sections 2.1 - 2.5.2 (CQF background) into new appendix ; former section 2.6 (Summary of TCQF benefits and goals for DetNet) is now section 2 and refers to that appendix; moved former section 6.1 (Solution Scope, Taxonomy and Status) into appendix ; moved former sections 6.2 (High-Speed Implementation) and 6.3 (Controller plane computation of cycle mappings) into new section preceding the remaining implementation considerations section. Not (yet) accepted: Aijun Wang's proposal to remove the pseudocode. Pseudocode with explanations has proven invaluable to implementers of RFCs such as , and the author(s) therefore think it should stay normative. TBD: Aijun Wang's proposal to simplify section 3 (Using TCQF in the DetNet Architecture and MPLS forwarding plane) - it is not yet clear what simplification is desired. Added Open Issues section, recording the renaming suggestion by Janos Farkas. Added Lou Berger, Aijun Wang, Donald Eastlake, Janos Farkas and Jinoo Joung to acknowledgements.

This document defines the IP header field, called the DS (for differentiated services) field. [STANDARDS-TRACK] This document defines a flexible solution for support of Differentiated Services (Diff-Serv) over Multi-Protocol Label Switching (MPLS) networks. [STANDARDS-TRACK] This document provides the overall architecture for Deterministic Networking (DetNet), which provides a capability to carry specified unicast or multicast data flows for real-time applications with extremely low data loss rates and bounded latency within a network domain. Techniques used include 1) reserving data-plane resources for individual (or aggregated) DetNet flows in some or all of the intermediate nodes along the path of the flow, 2) providing explicit routes for DetNet flows that do not immediately change with the network topology, and 3) distributing data from DetNet flow packets over time and/or space to ensure delivery of each packet's data in spite of the loss of a path. DetNet operates at the IP layer and delivers service over lower-layer technologies such as MPLS and Time- Sensitive Networking (TSN) as defined by IEEE 802.1. This document specifies version 6 of the Internet Protocol (IPv6). It obsoletes RFC 2460. This document specifies the Deterministic Networking (DetNet) data plane when operating over an MPLS Packet Switched Network. It leverages existing pseudowire (PW) encapsulations and MPLS Traffic Engineering (MPLS-TE) encapsulations and mechanisms. This document builds on the DetNet architecture and data plane framework. Futurewei Technologies USA This memo explain requirements, benefits and feasibility of a new DetNet service function tentatively called "flow interleaving". It proposes to introduce this service function into the DetNet architecture. Flow interleaving can be understood as a DetNet equivalent of the IEEE TSN timed gates. Its primary role is intended to be at the ingress edge of DetNet domains supporting higher utilization and lower bounded latency for flow aggregation (interleaving of flows into a single flow), as well as higher utilization and lower bounded latency for interleaving occurring in transit hops of the DetNet domain in conjunction with in-time per-hop bounded latency forwarding mechanisms. This document describes the use of RSVP (Resource Reservation Protocol), including all the necessary extensions, to establish label-switched paths (LSPs) in MPLS (Multi-Protocol Label Switching). Since the flow along an LSP is completely identified by the label applied at the ingress node of the path, these paths may be treated as tunnels. A key application of LSP tunnels is traffic engineering with MPLS as specified in RFC 2702. [STANDARDS-TRACK] This document describes extensions to Resource Reservation Protocol - Traffic Engineering (RSVP-TE) for the set up of Traffic Engineered (TE) point-to-multipoint (P2MP) Label Switched Paths (LSPs) in Multi- Protocol Label Switching (MPLS) and Generalized MPLS (GMPLS) networks. The solution relies on RSVP-TE without requiring a multicast routing protocol in the Service Provider core. Protocol elements and procedures for this solution are described. There can be various applications for P2MP TE LSPs such as IP multicast. Specification of how such applications will use a P2MP TE LSP is outside the scope of this document. [STANDARDS-TRACK] Bit Index Explicit Replication (BIER) is an architecture that provides optimal multicast forwarding through a "multicast domain", without requiring intermediate routers to maintain any per-flow state or to engage in an explicit tree-building protocol. When a multicast data packet enters the domain, the ingress router determines the set of egress routers to which the packet needs to be sent. The ingress router then encapsulates the packet in a BIER header. The BIER header contains a bit string in which each bit represents exactly one egress router in the domain; to forward the packet to a given set of egress routers, the bits corresponding to those routers are set in the BIER header. The details of the encapsulation depend on the type of network used to realize the multicast domain. This document specifies a BIER encapsulation that can be used in an MPLS network or, with slight differences, in a non-MPLS network. Segment Routing (SR) leverages the source routing paradigm. A node steers a packet through an ordered list of instructions, called "segments". A segment can represent any instruction, topological or service based. A segment can have a semantic local to an SR node or global within an SR domain. SR provides a mechanism that allows a flow to be restricted to a specific topological path, while maintaining per-flow state only at the ingress node(s) to the SR domain. SR can be directly applied to the MPLS architecture with no change to the forwarding plane. A segment is encoded as an MPLS label. An ordered list of segments is encoded as a stack of labels. The segment to process is on the top of the stack. Upon completion of a segment, the related label is popped from the stack. SR can be applied to the IPv6 architecture, with a new type of routing header. A segment is encoded as an IPv6 address. An ordered list of segments is encoded as an ordered list of IPv6 addresses in the routing header. The active segment is indicated by the Destination Address (DA) of the packet. The next active segment is indicated by a pointer in the new routing header. This document provides an overall framework for the Deterministic Networking (DetNet) data plane. It covers concepts and considerations that are generally common to any DetNet data plane specification. It describes related Controller Plane considerations as well. The Segment Routing over IPv6 (SRv6) Network Programming framework enables a network operator or an application to specify a packet processing program by encoding a sequence of instructions in the IPv6 packet header. Each instruction is implemented on one or several nodes in the network and identified by an SRv6 Segment Identifier in the packet. This document defines the SRv6 Network Programming concept and specifies the base set of SRv6 behaviors that enables the creation of interoperable overlays with underlay optimization. This document specifies a new architecture for the forwarding of multicast data packets. It provides optimal forwarding of multicast packets through a "multicast domain". However, it does not require a protocol for explicitly building multicast distribution trees, nor does it require intermediate nodes to maintain any per-flow state. This architecture is known as "Bit Index Explicit Replication" (BIER). When a multicast data packet enters the domain, the ingress router determines the set of egress routers to which the packet needs to be sent. The ingress router then encapsulates the packet in a BIER header. The BIER header contains a bit string in which each bit represents exactly one egress router in the domain; to forward the packet to a given set of egress routers, the bits corresponding to those routers are set in the BIER header. The procedures for forwarding a packet based on its BIER header are specified in this document. Elimination of the per-flow state and the explicit tree-building protocols results in a considerable simplification. This document describes the flow and service information model for Deterministic Networking (DetNet). These models are defined for IP and MPLS DetNet data planes. This document presents a timing model for sources, destinations, and Deterministic Networking (DetNet) transit nodes. Using the model, it provides a methodology to compute end-to-end latency and backlog bounds for various queuing methods. The methodology can be used by the management and control planes and by resource reservation algorithms to provide bounded latency and zero congestion loss for the DetNet service. Sangmyung University Huawei ZTE Corporation Futurewei Technologies This draft is to facilitate the understanding of the data plane enhancement solutions, which are suggested currently or can be suggested in the future, for deterministic networking. This draft provides criteria for classifying data plane solutions. Examples of each category are listed, along with reasons where necessary. Strengths and limitations of the categories are described. Suitability of the solutions for various services of deterministic networking are also mentioned. Reference topologies for evaluation of the solutions are given as well. Futurewei Technologies Inc. University of Tuebingen KOEVOO This memo describes per-packet stateless strict and loose path steered replication and forwarding for "Bit Index Explicit Replication" (BIER) packets (RFC 8279); it is called "Tree Engineering for Bit Index Explicit Replication" (BIER-TE) and is intended to be used as the path steering mechanism for Traffic Engineering with BIER. BIER-TE introduces a new semantic for "bit positions" (BPs). These BPs indicate adjacencies of the network topology, as opposed to (non-TE) BIER in which BPs indicate "Bit-Forwarding Egress Routers" (BFERs). A BIER-TE "packets BitString" therefore indicates the edges of the (loop-free) tree across which the packets are forwarded by BIER-TE. BIER-TE can leverage BIER forwarding engines with little changes. Co-existence of BIER and BIER-TE forwarding in the same domain is possible -- for example, by using separate BIER "subdomains" (SDs). Except for the optional routed adjacencies, BIER-TE does not require a BIER routing underlay and can therefore operate without depending on a routing protocol such as the "Interior Gateway Protocol" (IGP). Futurewei Technologies USA Stewart Bryant Ltd The purpose of this memo is to explain the challenges and limitations of existing (standardized) bounded latency queuing mechanisms for desirable (large scale) MPLS and/or IP based networks to allow them to support DetNet services. These challenges relate to low-cost, high-speed hardware implementations, desirable network design approaches, system complexity, reliability, scalability, cost of signaling, performance and jitter experience for the DetNet applications. Many of these problems are rooted in the use of per- hop, per-flow (DetNet) forwarding and queuing state, but highly accurate network wide time synchronization can be another challenge for some networks. This memo does not intend to propose a specific queuing solution, but in the same way in which it describes the challenges of mechanisms, it reviews how those problem are addressed by currently proposed new queuing mechanisms. Huawei Huawei Huawei Huawei China Mobile This document presents the overall framework and key method for Large-scale Deterministic Network (LDN). LDN can provide bounded latency and delay variation (jitter) without requiring precise time synchronization among nodes, or per-flow state in transit nodes. Huawei Huawei This document presents a queuing mechanism with multiple cyclic buffers. Huawei Huawei China Mobile Sangmyung University ETRI One of the goals of DetNet is to provide bounded end-to-end latency for critical flows. This document defines how to leverage Segment Routing (SR) to implement bounded latency. Specifically, the SR Identifier (SID) is used to specify transmission time (cycles) of a packet. When forwarding devices along the path follow the instructions carried in the packet, the bounded latency is achieved. This is called Cycle Specified Queuing and Forwarding (CSQF) in this document. Since SR is a source routing technology, no per-flow state is maintained at intermediate and egress nodes, SR-based CSQF naturally supports flow aggregation that is deemed to be a key capability to allow DetNet to scale to large networks. Futurewei Technologies USA Sympotech Independent ZF Friedrichshafen AG This memo proposes a mechanism called "guaranteed Latency Based Forwarding" (gLBF) as part of DetNet for hop-by-hop packet forwarding with per-hop deterministically bounded latency and minimal jitter. gLBF is intended to be useful across a wide range of networks and applications with need for high-precision deterministic networking services, including in-car networks or networks used for industrial automation across on factory floors, all the way to ++100Gbps country-wide networks. Contrary to other mechanisms, gLBF does not require network wide clock synchronization, nor does it need to maintain per-flow state at network nodes, avoiding drawbacks of other known methods while leveraging their advantages. Specifically, gLBF uses the queuing model and calculus of Urgency Based Scheduling (UBS, [UBS]), which is used by TSN Asynchronous Traffic Shaping [TSN-ATS]. gLBF is intended to be a plug-in replacement for TSN-ATN or as a parallel mechanism beside TSN-ATS because it allows to keeping the same controller-plane design which is selecting paths for TSN-ATS, sizing TSN-ATS queues, calculating latencies and admitting flows to calculated paths for calculated latencies. In addition to reducing the jitter compared to TSN-ATS by additional buffering (dampening) in the network, gLBF also eliminates the need for per-flow, per-hop state maintenance required by TSN-ATS. This avoids the need to signal per-flow state to every hop from the controller-plane and associated scaling problems. It also reduces implementation cost for high-speed networking hardware due to the avoidance of additional high-speed speed read/write memory access to retrieve, process and update per-flow state variables for a large number of flows. China Environment for Network Innovations (CENI) Translated with permission from chinese version at: https://ceni.org.cn/406.html IEEE 802.1 Working Group IEEE Time-Sensitive Networking (TSN) Task Group. IEEE Time-Sensitive Networking (TSN) Task Group.

Background

Cyclic Queuing and Forwarding (CQF) Cyclic Queuing and Forwarding (CQF) is a bounded (guaranteed) per-hop latency forwarding mechanism standardized for use in Ethernet switched networks by the IEEE TSN working group originally via (802.1 Qch), which later became annex T of . See also , Section 6.6. CQF is not a separate forwarding mechanism but simply a profile of the IEEE Time Aware Shaper (TAS) standard , which introduced Time-Gated Queues. CQF uses a two-queue based forwarding mechanism on every switch along a path between a sender and receiver. One queue is used to receive and store frames destined toward a particular outgoing interface on the switch, the other queue is used simultaneously to send frames to the same outgoing interface. At every cycle time T_c interval these two queues are swapped, or in terms of Time-Gated Queues, one is closed for sending, the other is opened for sending. This operation is synchronized across all switches in the network by network wide synchronized clocks, so that all queues open and close at the same time. For a path of h hops, the end-to-end latency bound is between (h-1) * T_c + DT and (h+1) * T_c. DT is the so-called dead time at the end of a cycle during which no frames can be transmitted from the sending queue to ensure that the last byte of the last frame will be received earlier than the end of the same cycle on the receiving switch. A core contributor to DT is the (physical) link between the sending and receiving switch. DT needs to be larger than the latency of this link, including physical propagation latency (speed of light), possible error correction latencies, and interface serialization latency. T_c needs to be chosen carefully: The larger it is, the higher the bounded latency. The smaller it is, the fewer bytes (and hence frames) will fit into a cycle. To admit flows into a CQF network, the ingress switch uses per-flow Time-Gated Queues. In the simplest case, such a gate is configured to admit a maximum number of bytes from the flow into every cycle. More advanced admission control can be performed for bursty flows. For example, N bursty flows f_i = 0...(N-1) could share admitted bandwidth by each having their burst admitted in different cycles c_i = c % N + c_i, where c is a continuous increasing cycle number.

Benefits of CQF with higher speed links Typical CQF deployments in manufacturing networks with 1Gbps links use cycle intervals of no less than hundreds of microseconds. In a network with a small diameter, say less than 8 hops, this is sufficient to provide an end-to-end latency bound on the order of several milliseconds. As link speeds increase from 100Mbps to 1Gbps, 10Gbps, 100Gbps or even higher in larger networks, either more bytes can be transmitted within the same cycle interval, or a smaller cycle interval suffices to transmit the same number of bytes per cycle as in low-speed networks. Likewise, serialization latency and hence DT decrease with higher speed links. Overall, this makes CQF more attractive for higher-speed networks than for lower-speed networks. shows a simple calculation of the number of bytes that can be transmitted in a cycle with different cycle intervals and link speeds. A minimum of 1500 bytes is labeled with * as a baseline because a typical maximum Ethernet frame is 1500 bytes and a selected cycle interval should at least allow one such frame size to be transmitted unless otherwise specified. TBD: These numbers probably need to be adjusted to reflect reducing DT based on serialization latency.

At 10Gbps, the cycle interval can be as small as 1.2 us if a 1500 byte frame needs to be transmitted in one cycle interval, and with 100Gbps links even 1 usec cycle time allows for 8 frames of 1500 byte each. These calculations are only approximate, because other factors also determine the cycle interval. However, it shows that as the link speed increases, the cycle interval can be greatly reduced in practice while satisfying the minimum amount of data transmitted in a single cycle. The end-to-end latency bound when applying CQF is determined by cycle interval and number of hops. Consequently, CQF with a smaller cycle interval has the potential to meet stricter end-to-end latency requirements in higher link speed networks or meet the same end-to-end latency requirement in networks with much larger network diameter (number of hops). Industrial automation has typical application period requirements, e.g., 100 us to 2 ms for isochronous traffic, 500 us to 1 ms for cyclic-synchronous and 2 to 20 ms for cyclic-asynchronous traffic. The network cycle interval is usually a fraction of the application period. When the cycle interval is in the order of tens of microseconds, CQF can be used to meet the strictest end-to-end latency requirements. For instance, assuming 24 hops and a cycle interval of 10us, the end-to-end latency bound is around (24+1)*10 = 250 us which has the potential to meet the latency bound requirement for isochronous traffic. In summary, a higher speed network makes the shorter cycle interval feasible because sufficiently large traffic volume can be transmitted within one cycle interval. A shorter cycle interval further offers shorter end-to-end latency and jitter bounds which provide CQF with the potential to meet stricter latency requirements in wider deployments while preserving its simplicity of latency calculation and provisioning. Therefore, there is a strong motivation to leverage CQF and at the same time to make cycle interval as short as possible.

Challenges of CQF with higher latency links Unlike the original targets of IEEE TSN work, DetNet targets not only IETF forwarding planes (IP, MPLS, ...), but also wide-area networks, and therefore longer physical propagation latencies. As shown in for fundamental (two buffer) CQF, the last byte sent by node A in cycle (i-1) has to be ready for sending at node B before the start of cycle i. To achieve this, a dead time (DT) is imposed: a time interval, usually at the end of a cycle, during which a node does not send scheduled CQF packets. Dead time is at least the sum of the maximum propagation delay to the next node, the maximum processing delay at the next node and the maximum other time variations. Therefore either the longer propagation or longer processing delay makes dead time larger. Packets from DetNet service are likely to be propagated over long links in the wider area. It takes around 5us per kilometer to propagate, i.e. 0.5ms every hundred kilometers. Hence the dead time can be as large as milliseconds or tens of milliseconds in case of links of hundreds of kilometers or longer and larger processing delays. That would make the dead time eat up most of the cycle interval when cycle interval is short (e.g., of the same order of magnitude as the dead time, or one order higher). The useful time in a cycle is then much reduced. In some extreme cases, when the link is long and the cycle interval is set to extremely short, the first packet sent in a cycle by a node cannot be received in the same cycle interval at the next node. That makes the useful time in a cycle reach zero in two buffer CQF. Two buffer CQF is then no longer usable. As a result of these considerations, TSN CQF networks are reasonably limited to at most a few km per hop, beyond which DT exceeds common cycle times and the possible throughput of CQF traffic hence drops to zero.

Time | | | | Node A | cycle i-1 | cycle i | cycle i+1 | | | | | Sending ---------------+---------------------------------- |+------+ |+------+ |+------+ | ||//////| ||//////| ||//////| | |+------+ |+------+ |+------+ | | buf_1 | buf_2 | buf_1 | | | | | | | | | | DT | | DT | | DT | Node B | |<--->| |<--->| |<--->| | | | | Receiving-------------------------------------------------- | +------+| +------+| +------+| | |//////|| |//////|| |//////|| | +------+| +------+| +------+| | buf_1 | buf_2 | buf_1 | | | | | | | | | Node B | | | | | | | | Sending -------------------------------------------------- | |+------+ |+------+ | | ||//////| ||//////| | | |+------+ |+------+ | | | buf_1 | buf_2 DT=Dead Time ]]>

Review of CQF benefits and challenges for DetNet In summary, CQF has a range of benefits for DetNet. It provides bounded latency. It provides tightly bounded jitter. It has a very simple and easily standardized calculus for its bounded latency and jitter. It has very simple per-hop forwarding machinery (cyclic queues) easily supportable in high-speed network equipment. Like Diffserv forwarding, it does not use per-hop,per-flow state in the forwarding plane and therefore does not require per-hop,per-flow signaling with the DetNet controller-plane, allowing it to scale to a large number of flows. The faster the links are, the lower the per-hop latency impact of the cyclic queuing mechanism. The core limitation of CQF, which TCQF intends to solve, lies in its use of the arrival time to determine the cycle into which the packet is to be placed, see for more details. Cycle times should be as short as feasible to support lower end-to-end latency (). When networks have longer links, or links with higher propagation jitter as in Metro and WAN, the dead time increases, which reduces the possible utilization or requires longer cycle times. When shorter cycle times are feasible because of higher speed links, this would require an increase in clock-synchronization accuracy.

Tagged CQF Tagging of CQF packets with cycle identifiers can be used to solve the aforementioned dilemma with minor changes to the fundamental two buffer CQF. This section introduces this mechanism with multiple buffers and CQF cycle identification in the packet header. From here on, this document uses the terms packet (as used for IP, MPLS and other IETF forwarding planes) and packet buffer, as opposed to frame as used by IEEE.

CQF with more than two buffers CQF can use more than two buffers to minimize the dead time and increase the useful time in a cycle so as to support long link delay. shows how a three buffer CQF works in a rotating manner in general. Node A sends packets in cycle (i-1). The time interval over which node B receives these packets spans two cycles, cycle (i-1) and cycle i. Hence a method is needed to make node B send them all at once in cycle (i+1), in order to ensure that packets sent in a single cycle by the previous node are always sent out in a single cycle by the current node.

Time | | | | Node A | cycle i-1 | cycle i | cycle i+1 | | | | | Sending ---------------+---------------------------------- |+----------+ |+----------+ |+----------+ | ||//////////| ||//////////| ||//////////| | |+----------+ |+----------+ |+----------+ | | buf_1 | buf_2 buf_3 | | | | | | | | | ->| |<- ->| |<- ->| |<- | DT DT DT | ------------------------------------------------- Node B | +-----------+ +-----------+ +-----------+ | |///////////| |///////////| |///////////| Receiving | +-----------+ +-----------+ +-----------+ | buf_1 | buf_2 | buf_3 | | | | | | | | | | | | | | | | | ---------------|---------------------------------- Node B | | |+----------+ |+----------+ | | ||//////////| ||//////////| Sending | | |+----------+ |+----------+ | | buf_1 buf_2 DT=Dead Time ]]>

More than three buffers will be required when the receiving interval at node B for packets sent in a single cycle interval from node A spans over more than two cycle interval boundaries. This can happen when the time variance (jitter) including propagation, processing, regulation, clock synchronization variance (so called Maximum Time Interval Error - MTIE) and other factors between two neighbouring DetNet nodes can become larger than a single cycle time.

From CQF with multiple buffers to TCQF Note that due to the variance in time, the receiving interval at the downstream node can be much larger than one cycle interval in which the upstream node transmits. When time variance is large and cycle interval and dead time are set small, the possible receiving time of the last few packets from node A’s cycle (i-1) at node B can overlap with the possible receiving time of the first few packets from node A’s cycle i in different rounds of buffer rotations. Hence, when the buffer number is larger than two, if the receiving side still uses the traditional CQF implicit time borderline to demarcate the receiving packets from the consecutive cycles of the upstream node, this may cause ambiguity in identifying the correct sending cycle of the upstream node, and hence in deciding into which output buffer of the current node the received packets have to be placed. shows such an ambiguity when time based cycle demarcation is used. The packet sent by node A in its cycle (i-1) can be received at any time in the receiving interval indicated as “receiving window for A’s buf_1” in Figure 4. The receiving window refers to the time interval between the earliest time that the first packet sent in a given cycle from an upstream node is processed and enqueued in an output buffer and the latest time that the last packet of the cycle is processed and enqueued in an output buffer. Network operators may configure the size of the receiving window, taking the time variance of their networks into account. It can be seen that the spanning time period of receiving window is longer than the cycle interval. This is because there is a large time variance experienced between A and B, e.g. varying processing time for different packets in different cycles. This does not mean that the receiving interval of every cycle always spans such a large receiving window. The receiving window is determined by the worst-case time variance, and that value has to be used for time-based cycle demarcation.

Time | | | | | Node A | cycle | cycle | cycle | cycle | | i-1 | i | i+1 | i+2 | Sending ----------+--------+--------+--------+ |+-----+ |+-----+ |+-----+ |+-----+ | ||/////| ||/////| ||/////| ||/////| | |+-----+ |+-----+ |+-----+ |+-----+ | | buf_1 | buf_2 | buf_3 | buf_4 | | | | | | | | | | | ->| |<- ->| | ->| | ->| | | DT DT DT DT | -------------------------------------- | +-----------+receiving window Node B | |///////////|for A's buf_1 | +-----------+ Receiving | put to B's buf_1 | | ->| |<- ambiguity window 1 | | +-----------+receiving window | |///////////|for A's buf_2 | | +-----------+ | | put to B's buf_2 | | | | ->| |<- ambiguity window 2 | | | | | | +-----------+receiving window | | | |///////////|for A's buf_3 | | | +-----------+ | | | put to B's buf_3 | | | | | | .......... | | | -|--------|--------|--------|--------------- Node B | | | | | | | | | +-----+|+-----+ |+-----+ |+-----+ Sending | | | |/////|||/////| ||/////| ||/////| | | | +-----+|+-----+ |+-----+ |+-----+ | | | buf_4 | buf_1 | buf_2 | buf_3 DT=Dead Time ]]>

When a packet is received in ambiguity window 1 in , node B is not able to use the receiving time to determine which buffer is the correct one to put the packet in because it cannot tell if the packet is sent from cycle (i-1) or cycle i on node A. If node B puts the packet into the wrong output buffer, the packet may experience unexpected delay. At the same time, the packet occupying the non- designated buffer may violate the service contract between the end hosts and the DetNet network, with unpredictable consequences. The DT could be increased substantially to cover the time variance, so that the receiving windows do not overlap and the ambiguity is removed. However, it is not always practical and usually not desired because large DT will eat useful cycle time and cause the low utilization issue illustrated in . Therefore, it is desirable to keep DT as small as possible and at the same time identify the cycle interval correctly. With tagged CQF, the sending router A encodes the sending cycle identification in some existing or new packet header field as specified later in this document. This allows the receiving router B to determine the correct output port cycle buffer to place the data packet into. Except for the need for the operator to pre-configure this mapping on router B, based on the latency and jitter of the link (and of processing between the sending and receiving router) described above, tagging does not change the fundamental mechanism and benefits of CQF. Compared to CQF with multiple buffers, Tagged CQF can operate with clock synchronization of significantly lower accuracy than CQF. In CQF, the MTIE is an addend determining DT and should hence typically be less than 1% of the cycle time. In TCQF it is an addend of the permitted receive window and can hence, for example, be as large as the cycle time, i.e., 100 times larger. A network using TCQF with 100Gbps interfaces can hence use the same or less expensive clock synchronization setup than a CQF network with 1Gbps interfaces. In addition, when the conditions of network connections change, network operations can dynamically change the mappings. CQF with multiple buffers but without tagging has been proposed to IEEE TSN in , but has not been adopted. Instead of relying on a cycle tag in a packet header, it still relies solely on the arrival time of packets, and hence cannot resolve arrival time ambiguities as TCQF can, because it does not know the cycle from which the packet was sent.

Solution Scope, Taxonomy and Status for DetNet adoption considerations

Solution scope TCQF is proposed not as a standalone solution, but in conjunction with as a per-flow mechanism to be used on the ingress TCQF router to allow more flexible per-flow management of admission of flow bursts into cycles than presented in this document. The reason for splitting up the proposal into two documents is because can be combined with all "mostly-synchronous" per-hop mechanisms, for example beyond TCQF also (of course) CQF and GLBF ().

Taxonomy Using the section number/titles of the taxonomy document the TCQF technology can be classified as follows. 4.1 Per Hop Dominant Factor for Latency Bound The per-hop dominan latency bound is Category 3. The per-hop latency for each class is determined by the cycle size, which has to be determined by the sum of burst sizes required by the flows. This burst can be split up across multiple cycles by the mechanisms described in for flows whose rate is low enough that they do not need resources in every cycle. 5.1 Periodicity TCQF is Periodic, however, by using tagging it reduces the required synchronization across the network (using CQF as a reference) by potentially a factor of 100 or more, allowing to use the same or even less stringent clock synchronization at 100 times or more speedup versus current CQF networks. 5.2. Traffic Granularity TCQF has Class level granularity. In its current form, this draft only describes operation for a single cycle class, but it can easily be extended to use multiple classes. In addition, additional differentiation into more classes can happen at the ingress edge through . However, those additional options do not increase the amount of per-hop state in the TCQF network, that will stay limited to the cycle states required for the per-hop classes (e.g.: just one class now). Hence TCQF maintains minimal state per-hop. 5.3. Time Bounds TCQF is Bounded (Left and Right bounded) as determind by the time in which the cycle the packet is in is sent. In result, The total end-to-end jitter of a packet is limited to the cycle time deployed, as is the per-hop jitter. 5.4 Service Order TCQF service order is solely arrival based. Suitable Categories for DetNet TCQF is "6.3. Class level periodic bounded category" according to the categorization.

Status TCQF has a high-speed (100Gbps interface) ASIC/FPGA implementation which was deployed and tested by an academic network test organization; a test report is in . There is also a large-scale simulation of the algorithm/mechanisms and results presented at a peer-reviewed academic conference, see .

CSQF (CSQF) describes a variation of the cyclic queuing mechanism in which the cycle identifier is not mapped by a mapping table in each node (as in TCQF); instead, the packet header carries the sequence of cycles for every cyclic queuing hop. In the draft, this is proposed specifically for networks using Segment Routing and can therefore allocate N SIDs for N cycles, one per cycle, so that a SID sequence in the header indicates for each hop which cycle to use. The core new functionality enabled by eliminating the cycle mapping table on the routers and moving the sequence of cycles into the header is the ability to utilize in a flexible fashion more than a fixed number of cycles, independently on each hop. Assume a minimum of N (e.g.: N = 3) cycles would be required in a particular deployment with TCQF. If CSQF is then set up with e.g.: N + 4 = 7 cycles, then it would be possible for the controller-plane to delay packets of a flow on every hop by 1,2,3 or 4 more cycles than necessary at minimum. This can make it easier to achieve higher utilization when the controller plane manages a large number of flows in large-scale DetNets and does not allocate bandwidth to every flow in every cycle. This naturally leads to uneven utilization of cycles and the problem of managing distribution of traffic load across cycles. discusses this overall advanced controller-plane traffic management and how different queuing options can be used in such a setup. It also describes the necessary ingress processing to allow forwarding traffic flows only in such well engineered specific cycles. While such advanced cycle engineering may at first look quite complex, it should simply be compared to the mechanisms that already are standard in service provider networks to manage bandwidth/capacity by engineering per-flow paths across topologies with non-equal cost paths. In that overall complex problem space, managing distribution of traffic across cycles is but a minor extension. Note that TCQF can of course also benefit from such advanced cycle engineering at the controller plane, albeit less flexibly than CSQF. Given how CSQF and TCQF share all the forwarding behavior except for where the cycle Identifier is retrieved from and how it is mapped, it would also be very useful to consider both approaches as options of a single target standard. It seems unlikely, though, that an implementation that can support TCQF could not support CSQF - or vice versa.

TCQF with multiple priorities TSN CQF permits establishing multiple independent cyclic queuing instances, which creates more flexibility. Consider likewise that in DetNet there are separate packet header fields for packet priority and cycle identifier. For each priority, a separate instance of TCQF is established, and the priority decides which instance of TCQF the packet gets processed by, whereas the cycle identifier determines the cycle within the TCQF instance. Consider for example a setup with 4 priorities 1..4. The cycle time for the highest priority 1 is C. The cycle time for priority 2 is 2 * C, for priority 3 3 * C and for priority 4 4 * C. With strict priority queuing, packets from a priority 1 cycle queue will always be sent over those from priorities 2...4, and so on. As a result, a flow can now be given one out of 4 priorities, each with an increasing per-hop latency: C (prio 1), 2C (prio 2), 3C (prio 3), 4C (prio 4). This of course also requires admission control not to allow full utilization of the capacity of cycles in each class. In a simple static splitting of capacity across classes, each cycle of each priority could for example be allowed to be utilized up to 25%. This multi-priority "extension" to TCQF is in this version of the document only mentioned as an appendix, because it is not clear if this degree of flexibility is desired in a first-generation target standard for TCQF. Given how both priority and cycle identifiers are needed, this mechanism would certainly require a new extension header for both MPLS and IP/IPv6, such as the one proposed in this document to carry the Cycle Identifier, and then the priority could be indicated by the IP header DSCP.

TSN Multiple Buffer CQF CQF with multiple buffers but without tagging has been proposed to IEEE TSN in , but has not been adopted. Instead of relying on a cycle tag in a packet header as proposed in this memo, it still relies solely on the arrival time of packets, and hence cannot resolve arrival time ambiguities as TCQF can, because it does not know the cycle from which the packet was sent, or the cycle for which it is intended. Consider that multiple buffer CQF is like TCQF, except the cycle id is missing from the packet that is sent. Upon arrival at the receiving router, the sending cycle ID has to be determined solely by the time the packet is received (reception timestamp) because this time is an indicator of the sending timestamp and hence the sending cycle. The sum of MTIE, processing variation, link propagation latency and other variations from layer 1 and layer 2 processing (forward error correction, retransmissions) is the error of the sending time that the receiving router can determine. As soon as this error becomes so large that the receiving router cannot unambiguously determine a sending cycle, the mechanism does not work anymore. The receiving router can also not simply assume that a packet was sent in one of the possible cycles, because when this is not the actual sending cycle, then such an assumption will cause possible overruns of cycle buffers and hence failure of admission control and packet drops or congestion. In result, multiple buffer CQF without carrying a target cycle in a packet header seems not feasible to actually solve the issue of real propagation latency variation in transmission, or the perceived variation in propagation due to jitter in clocks between adjacent nodes. https://www.iana.org/assignments/ipv6-parameters/ipv6-parameters.xhtml