]> Futurewei Technologies USA

2220 Central Expressway Santa Clara CA 95050 USA tte@cs.fau.de

Sympotech

CA USA ludwig@clemm.org

Independent

UK sb@stewartbryant.com

Rohde & Schwarz

DE stefan.hommes@icloud.com

DETNET This memo proposes a mechanism called "guaranteed Latency Based Forwarding" (gLBF) as part of DetNet for hop-by-hop packet forwarding with per-hop deterministically bounded latency and minimal jitter. gLBF is intended to be useful across a wide range of networks and applications that need high-precision deterministic networking services, from in-car networks and networks for industrial automation on factory floors all the way to country-wide networks at 100 Gbps and beyond. Unlike other mechanisms, gLBF does not require network-wide clock synchronization, nor does it require per-flow state on network nodes. It thereby avoids the drawbacks of other known methods while retaining their advantages. Specifically, gLBF uses the queuing model and calculus of Urgency Based Scheduling (UBS, ), which is used by TSN Asynchronous Traffic Shaping . gLBF is intended as a plug-in replacement for, or a mechanism in parallel to, TSN-ATS, because it allows reuse of the same controller-plane design that selects paths for TSN-ATS, sizes TSN-ATS queues, calculates latencies and admits flows onto the calculated paths with the calculated latencies. In addition to reducing jitter compared to TSN-ATS through additional buffering (dampening) in the network, gLBF also eliminates the per-flow, per-hop state maintenance required by TSN-ATS. This avoids having to signal per-flow state from the controller-plane to every hop, and the associated scaling problems. It also reduces the implementation cost of high-speed networking hardware, because it avoids the additional high-speed read/write memory accesses needed to retrieve, process and update per-flow state variables for a large number of flows.

Overview (informative)

Terminology

CaaS

Control as a Service. Cloud (compute) and network services that enable control (loops) for network-connected devices, for example cars.

CQF

Cyclic Queuing and Forwarding. A queuing mechanism defined in Annex T of IEEE 802.1Q.

DT

Dead Time. A term from CQF indicating the time during each cycle in which no frames can be sent because the receiving node could not receive them into the desired cycle buffer.

node

A system that, with respect to gLBF, does not act as a host (sender/receiver). This memo avoids the term router so as not to imply an IP/IPv6 router as opposed to an LSR (label switching router). Likewise, a node can also be an IEEE 802.1 bridge implementing gLBF.

TCQF

Tagged Cyclic Queuing and Forwarding. A mechanism proposed to DetNet, see .

Summary This memo proposes a mechanism called "guaranteed Latency Based Forwarding" (gLBF) for hop-by-hop packet forwarding with per-hop deterministically bounded latency and minimal jitter. gLBF is intended to be useful across a wide range of networks and applications that need high-precision deterministic networking services, from in-car networks and networks for industrial automation on factory floors all the way to country-wide networks at 100 Gbps and beyond. At its foundation, gLBF addresses the problems of burst accumulation and jitter accumulation across multiple hops to achieve per-hop fixed latency guarantees. This allows easy calculation of end-to-end latency as the sum of these per-hop latencies. In particular, gLBF eliminates the need for the controller-plane to interact with the intermediate-hop routers of a gLBF path. This simplifies the controller-plane and network operations, and increases the number of flows the network can support: with gLBF, per-flow state is only required on ingress routers. Although the controller-plane in general is out of scope for this document, discusses aspects relevant to system design with gLBF, including the opportunity for a decentralized controller-plane/admission control (). Burst accumulation is the phenomenon whereby bursts of packets from senders in an admission-controlled network grow across intermediate nodes. It can only be managed with exponential complexity in admission control processing and a significant increase in worst-case end-to-end latency and/or lower maximum utilization. Dynamic, large-scale or easy-to-manage admission control solutions need forwarding mechanisms without this problem, so that admission control for bandwidth and jitter/buffer requirements can be linear: decomposed into purely per-hop calculations that are independent of changes in prior-hop traffic characteristics. This is not possible without forwarding-plane solutions that overcome burst accumulation. Existing solutions address burst accumulation by maintaining inter-packet gaps on a per-flow basis, as in TSN Asynchronous Traffic Shaping (TSN-ATS). gLBF instead ensures that inter-packet gaps are maintained without the need for per-flow state. The basic idea is to assign a specific queuing delay budget to each node and traffic class. This budget is known in advance from the admission control calculus. When the packet is transmitted, the queuing delay it actually experienced in the node is subtracted from that budget, and the result is carried in a new header field of the packet. On receipt, the next node passes the packet through a delay stage, where the packet waits for the time indicated in that field before the node proceeds with regular processing. In this way, queuing delay variations are absorbed, and deterministic delay is achieved without the possibility of burst accumulation. By addressing burst accumulation, gLBF also overcomes the problem of jitter accumulation, the second core problem of mechanisms such as TSN-ATS: depending on the amount of competing admitted traffic on a hop at any point in time, packets of a flow may experience anything from zero to the maximum delay across the hop. This is the per-hop jitter. It adds up across multiple hops, so applications requiring (near-)synchronous packet delivery cannot rely solely on such mechanisms. It likewise limits how highly networks with a large number of bounded latency flows can be utilized. The basic principle on which gLBF operates was proposed by researchers in the early 1990s under the name "Dampers". Dampers were not pursued or adopted at the time because network equipment lacked the necessary capabilities. Only recent and upcoming improvements in advanced forwarding planes make it possible to build them at scale and at acceptable cost. Unlike other proposals, gLBF does not require network-wide clock synchronization. Likewise, it does not need to maintain per-flow state on network nodes, because the delay budget and the queuing delay variations to be absorbed are carried in the packets themselves, making them "self-contained". This eliminates the per-flow, per-hop state maintenance required by TSN-ATS, with its scaling problems of signaling per-flow state from the controller-plane to every hop, and with the cost, in high-speed networking hardware, of the high-speed read/write memory accesses needed to retrieve, process and update these per-flow state variables for a large number of flows. Unlike other damper proposals, gLBF also supports the queuing model and calculus of Urgency Based Scheduling (UBS, ), which is used by TSN-ATS. gLBF is intended as a plug-in replacement for, or a mechanism in parallel to, TSN-ATS, because it allows reuse of the same controller-plane design that selects paths for TSN-ATS, sizes TSN-ATS queues, calculates latencies and admits flows onto the calculated paths with the calculated latencies. While gLBF as presented here is intended for use with IETF forwarding protocols, to provide DetNet QoS with bounded latency and lower, bounded jitter, it would be equally applicable to other forwarding planes, such as IEEE 802.1 Ethernet switching, assuming appropriate packet headers are defined to carry the hop-by-hop and end-to-end metadata required by the mechanism.

Application scenarios and use cases gLBF addresses the same use cases that are targeted by deterministic networking and high-precision networking services in general. Common requirements of those services include the need to provide service levels within very tightly defined bounds, in particular very specific latencies without the possibility of congestion-induced loss. The ability to provide services with corresponding service level guarantees enables many applications that would simply not be feasible without such guarantees. The following describes some use cases and application scenarios. The development towards autonomous vehicles leads to new requirements and a high demand for computational resources that are often not available within a car. One solution is to reduce in-car processing to a minimum and to offload computationally expensive tasks to the cloud. Because of the safety implications of these use cases, delays in the transport of messages from the cloud to the car and vice versa are often not acceptable. This includes application scenarios such as trajectory planning for driverless vehicles, but also emergency notifications to surrounding cars, which require fast delivery of messages to avoid delayed reactions in case of an accident. While the use of TSN for in-vehicle networks has already been investigated and is more mature, real-time communication with guaranteed latencies between vehicles and the cloud is still an open challenge. Another important use case for the automotive industry is to further optimize the manufacturing process by taking more data sources into account. Since most SCADA systems and PLCs cannot connect to large data lakes or perform more advanced computational jobs, real-time communication between a cloud instance and the shop floor has several benefits. First, it integrates more data into the decision-making process, since control algorithms for automating manufacturing sites that run in a cloud environment can take more variables into account than single-plant control systems. Second, it reduces the resources needed on a particular factory floor or plant by transferring complex, recurring and resource-intensive computational jobs to a cloud provider, where computational power is not the limiting factor. However, in such cases it is important that the associated control can still occur in real time and according to very precise timing constraints. Such a development is already foreseen by trends such as Industry 4.0 and Control-as-a-Service (CaaS).

Background The following background introduces and explains gLBF step by step. It uses IEEE 802.1 "Time Sensitive Networking" (TSN) mechanisms for reference, because at the time of writing, TSN bounded latency mechanisms are the most commonly understood and deployed mechanisms providing bounded latency and jitter services. All mechanisms compared here, as well as those used by TSN, are based on an overall service design in which traffic flows have a well-defined rate and burstiness, which are tracked by the controller-plane and are called the "envelope" of the flow in this memo. The traffic model used for gLBF is taken from UBS. It gives the flow a rate r [bits/sec] and a burst size b [bits], and the traffic envelope condition is that the total number of bits w(t) of the flow over a period t [sec] must be equal to or smaller than (r * t + b). In one typical case, a flow sends one packet of size b every interval p [sec]. This translates into a rate r = b / p for the flow because, after the flow has sent the first packet of b bits, it takes p seconds until (r * t) reaches b again: (r * t) = ((b / p) * p). The controller-plane uses this per-flow information to calculate for each flow a path with sufficient free bandwidth and per-hop buffers so that the bounded end-to-end latency of packets can be guaranteed. It then allocates bandwidth and buffer resources to the flow so that further flows will not impact it. Within this framework, bounded latency mechanisms can in general be divided into "on-time" and "in-time" mechanisms.

In-time mechanisms "In-time" bounded latency mechanisms forward packets in an (almost) work-conserving manner. When there is no competing traffic in the network, packets of traffic flows that comply with their admitted envelope are forwarded without any queuing latency introduced by the mechanism. When the maximum amount of admitted traffic is present, packets of admitted flows experience the maximum guaranteed, so-called bounded, latency. As a result, in-time mechanisms introduce the maximum amount of jitter, because the amount of competing traffic can change quickly, and with it the latency of packets. IEEE TSN Asynchronous Traffic Shaping is the prime example of an in-time mechanism. IETF Integrated Services () is an older mechanism based on the same principles. In-time mechanisms have the benefit of not requiring clock synchronization between nodes for their queuing.

On-time mechanisms On-time bounded latency mechanisms deliver packets (near-)synchronously with zero or a small maximum jitter, significantly smaller than that of in-time mechanisms. IEEE TSN Cyclic Queuing and Forwarding (CQF) is an example of an on-time mechanism, as is the Tagged Cyclic Queuing and Forwarding (TCQF) mechanism proposed to DetNet. Unlike the in-time mechanisms mentioned above, these mechanisms require clock synchronization between routers.

Control loops vs. in-time and on-time One class of applications that requires or prefers on-time (low-jitter) delivery of packets is control loops in vehicles or industrial environments, and hence in low-speed, short-range networks. Emerging or future use cases such as remote PLC or remote driving extend these requirements into metropolitan-scale networks. In these environments, on-time forwarding is also called synchronous forwarding for synchronous control loops. Typically, in synchronous control loops, central units such as Programmable Logic Controllers (PLC) control a set of sensors and actuators, polling or periodically receiving status information from sensors and sending action instructions to actuators. When packet forwarding is on-time (synchronous), the central unit knows exactly when sensors sent a packet and when packets are received by actuators, and can act accordingly. These solutions do not require sensors and actuators to have accurate, synchronized clocks. Instead, the central unit can control the time at which sensors and actuators perform their operations, within the accuracy of the (zero/low) jitter of the network packet transmission. When bounded latency forwarding is (only) in-time, edge nodes in the network and/or sensors and actuators need to convert the packets arriving with high jitter into an on-time arrival model, to continue supporting the model the application requires. This conversion is typically called a playout buffer mechanism. It requires time synchronization between senders and receivers, and hence most commonly requires the network to support clock synchronization for these edge devices and/or senders and receivers. As a result, existing bounded latency mechanisms that support synchronous, on-time delivery of packets require clock synchronization across the network: in on-time mechanisms like CQF for the forwarding mechanism itself, and in in-time mechanisms to synchronize edge devices and hosts.

Challenges with network wide clock synchronization Clock synchronization is a well-understood technology. However, it is also a significant operational cost factor, and potentially an equipment cost factor, for example through the difference in hardware cost between local oscillators of different accuracy in network devices. Clock synchronization, e.g., with IEEE 1588 PTP, is only deployed where no simpler solution exists that provides the same benefits without it. Today, for example, this means that in mobile networks only the so-called fronthaul deploys clock synchronization, but not the backhaul. As a result, it could be challenging to introduce new applications such as the remote PLC or remote driving applications mentioned above if they had to rely on a bounded latency network service with today's strict clocking requirements. But even in existing markets such as in-car or industrial networks, removing or reducing the need for clock synchronization could be a significant step towards lower cost and simpler solutions. This reduction in clock synchronization requirements is what gLBF provides.

gLBF introduction (informative)

Dampers The principle of the mechanism presented here is the so-called "Damper" mechanism, first described in the early 1990s but never standardized, primarily because the required forwarding was considered too advanced for the equipment of the time. These limitations are now starting to be resolved, so it seems like a good time to reintroduce the mechanism. The principle of damper-based forwarding is easily explained: when node A sends a packet, it has measured the latency L, i.e., how long the packet was processed in the node. The main component of L is the queuing latency of the packet in A caused by competing traffic to the same outgoing interface. Based on the admission control and queuing algorithms used in the node, there can be a known upper bound M(ax) for this processing latency. When the packet then arrives at the next-hop node B, B simply delays the packet further by (M-L). As a result, the packet has been forwarded synchronously from A to B with a constant latency of M(ax).

| ]]>

shows a single hop from node A to node B without Dampers. A receives a packet. The F(orwarding) module determines the outgoing interface towards node B, where the packet is enqueued into Q, because competing packets may already be waiting to be sent from Q to B. This happens because packets may arrive simultaneously from multiple input interfaces on A (not shown in the figure), and/or because the speed of the receiving interface on A is higher than the speed of the link to B. Forwarding F can be L2 switching and/or L3 routing; the choice does not impact the considerations described here. When the packet is finally sent from Q over the link to B, B repeats the same steps towards the next node (not shown). x is the reception time of the packet in A and B. The per-hop latency of the packet is xB - xA. It is predominantly determined by the time the packet has to wait in Q, plus any other relevant fixed or variable processing latency from F, plus the latency of the packet across the link. The latter is predominantly determined by the serialization latency of the packet plus the propagation latency of the link, which depends on the speed of light in the material used; in fiber, for example, light travels 31 percent slower than in vacuum. All factors impacting the hop A/B latency other than Q in A are naturally bounded: a well-defined maximum can be calculated or conservatively estimated. The transmission latency of the packet from A to B is composed of the serialization latency of the packet, which can be calculated from the packet length and the bit rate of the interface, and the propagation latency through the link, which can be calculated from the speed of light in the link material and the length of the link. To guarantee bounded latency through Q, an admission control system is required that tracks, accounts for and limits the total amount of traffic through Q. Admission control mechanisms rely on knowing the maximum burst that each traffic flow can cause, and add up those bursts to determine the maximum amount of simultaneous packet data in Q, which in turn determines the maximum latency of each individual packet through Q. With such an admission control system, one can therefore calculate a maximum hop A/B latency MAX for a packet. However, packets will naturally have a variable hop A/B latency lower than MAX, due to differences in packet size and in competing traffic. As a result, their relative arrival times xB in B will differ from their relative arrival times xA in A. This leads to so-called "burst accumulation", and hence to the problem that the admission control system cannot calculate the maximum burst and latency through Q in B as easily as it can for Q in A. If, however, packets could be forwarded such that their hop A/B latencies were all the same (synchronous, on-time), then the admission control system could apply the same calculus to B as to A. This is what damper mechanisms aim to achieve.

| |<--A/B on-time latency ------->| ]]>

shows the Damper mechanism that is simplest to explain, but not to implement. Node A measures the time xA and sends this value in a packet header to B. B measures the time xB directly on reception of the packet. It then delays the packet for a time (MAX-(xB-yA)). As a result, the latency (yB-yA) will be exactly MAX, up to the accuracy of the damper. The first challenge with this simple approach is the need to synchronize the clocks of A and B, so that (xB-yA) can be calculated correctly.

| |<--A/B on-time latency ------->| ]]>

shows how this can be resolved by also measuring the time at z. A calculates d = (MAX1-(zA-yA)) and sends d in a header of the packet to B. B then delays the packet in D by d relative to the locally measured time xB. Because neither the calculation of d in A nor the delay by d in D depends on clock synchronization between A and B, this approach eliminates the need for clock synchronization. As a consequence of this change, however, the latency incurred by transmitting the packet over the link is no longer included in the measurement. In this approach, MAX1 is the bounded latency for all processing of a packet except the link propagation latency P, which is the length of the medium divided by the speed of light in the medium, and MAX = MAX1 + P. The serialization latencies across the sending interface on A and the receiving interface on B can, however, be included in MAX1. It is only necessary for the measured timestamps zA and xB to logically refer to the same point in time, assuming a link of minimal length, e.g., a back-to-back connection. For example, the reference time could be the time when the first bit of the packet can be observed on the shortest wire between A and B. A can most likely measure zA only some fixed time offset o before that reference time, and hence needs to correct zA += o before calculating d. Likewise, B may only be able to measure xB after the whole packet has arrived. This would be l * r later than the reference time, where r is the serialization rate of the receiving interface on B and l is the length of the packet. B would hence need to adjust d -= l * r to subtract this time from the time the packet needs to be delayed in D.

| |<- Dampened Q ->| |<- A/B in-time latency ->| |<--A/B on-time latency ------->| ]]>

shows a further refined model for simpler implementation. Larger routers/switches are typically modular, and forwarding happens on a different modular component (such as a linecard) than the queuing for the outgoing interface. Requiring clock synchronization even within such a large device is undesirable. Therefore, shows damper operations occurring solely before enqueuing packets into Q and after dequeuing them. The Damper module measures the timestamp x, extracts d from the packet header, adjusts it according to the considerations described above, delays the packet by that value and then enqueues it. After the packet is dequeued, the Marking module measures the time z, adjusts it according to the same considerations, calculates the value of d and rewrites it in the packet header before sending the packet.

Dampers and controller-plane

While the damper mechanism described so far can be realized with different queuing mechanisms, and hence different calculi, on every interface that supports bounded latency, the role of the controller-plane involves other components besides admission control, and those need to be tightly coupled with admission control for efficient use of resources. illustrates the most common problem. For a new traffic flow from Src to Dst, the controller-plane needs to provide a path through the network and reserve the resources along it. The simplest form, for bandwidth reservation only, is called Constrained Shortest Path First (CSPF). When end-to-end latency guarantees also need to be provided in support of bounded latency, not only does the path calculation become more complex, but many per-hop bounded latency queuing mechanisms also support more than one per-hop latency on each hop, and the controller-plane needs to determine which one is best for each hop of a candidate path. As a result, to perform these operations, the controller-plane needs to know exactly which parameters the bounded latency queuing mechanism on each hop/interface supports. Standardizing these parameters ultimately means standardizing the bounded latency queuing mechanism, and not only the damper part.

gLBF specification (normative)

Damper with UBS queuing/calculus Guaranteed Latency Based Forwarding (gLBF) uses the queuing model of Urgency Based Scheduling (UBS), which is also used in TSN Asynchronous Traffic Shaping (TSN-ATS). This allows one and the same controller-plane, including its optimization algorithms for bandwidth and latency control, to be reused or co-developed for both TSN-ATS (or a DetNet L3 equivalent) and gLBF. Hence, gLBF should provide the most easily operationalized on-time solution for networks that want to evolve from an in-time model based on TSN-ATS, or that even want to operate both options in parallel. Effectively, gLBF replaces the per-flow interleaved regulators of UBS with per-flow stateless damper operations. Whereas UBS only provides in-time latency guarantees, gLBF as a result provides on-time latency service, but with fundamentally the same calculus as UBS.

| | Prio |-> ... -| Priority | |---> | | enque | \ +--------------+ | Schedule | | | +-------+ \-| Prio 8 queue |---| | | | +--------------+ +----------+ | +-----------------------------------------------+ ]]>

Strict priority scheduling always takes the next packet from the highest-priority queue (1 is highest) that has a pending packet, and forwards it. UBS defines two options for the traffic model of traffic flows. In the more flexible one, each flow specifies a rate r and a maximum burst size b (in bits). As a result, the bounded latency of a packet in priority 1 is (roughly) the latency required to serialize the sum of the bursts of all the flows admitted into priority 1. The bounded latency of a packet in priority 2 is the priority 1 latency plus the latency required to serialize the sum of the bursts of all the flows admitted into priority 2, and so on.

gLBF processing The following text extends/refines the damper processing as necessary for gLBF.

IIF| +-+ +-+ +-+ +-+ |OIF IIF| +-+ +-+ +-+ +-+ |OIF +------------------------+ +------------------------+ |<- Dampened Q ->| |<- Dampened Q ->| |<- A/B in-time latency -->| |<--A/B on-time latency -------->| ]]>

The Damper module (D) retrieves the delay from a packet header field. Requirements for that header field are defined in . Because serialization speeds differ between Incoming InterFaces (IIF), and because D will likely need to compensate the measured time x based on the packet length and serialization speed, an internal packet header should carry this information. Note that this is solely an implementation consideration and should not impact the configuration model of gLBF. The Queuing module is logically as described in UBS, except that the priority of a packet in gLBF is not selected based on per-flow state, but from an appropriate packet header field, which is looked up in the Packet Prio Enque stage of Q; the packet is then enqueued according to that field's value. Possible options for indicating such a priority in a packet header field are defined below in . Like Q, M needs to look up the packet priority to determine MAX1 for the packet, and hence calculate the value d that it rewrites in the packet header. The overall configuration data model to be defined for gLBF is hence the configuration data model of UBS, which is a set of priority queues and their maximum sizes, plus, for gLBF, a maximum latency value MAX1 for each of these queues, calculated by the controller-plane and used by the M module. Given that the node knows the serialization speed of the OIF, MAX1 for each queue could be derived automatically from the maximum length in bytes of each UBS priority queue. Additional configuration parameters to fine-tune the latency, such as increasing it by some percentage as a safety margin, might be beneficial though. Such implementation details are outside the scope of this specification. TBD: sketch a small YANG-like data model, even though it is trivial, as in the TCQF draft. It should primarily include additional diagnostic elements, such as the maximum occupancy of each priority queue and the number of overruns.

Error handling A well-specified standard for a damper mechanism such as the one described in this document also needs to handle error cases. The primary error condition is when the sending node A of a gLBF packet recognizes that the packet was enqueued for longer than MAX1, so that the delay to be written into the packet would have to be < 0. In this case, error signaling such as ICMP/ICMPv6 needs to be triggered (rate-limited!), and the packet needs to be discarded, to avoid admission control failures or congestion for other packets further down the path. The data model below describes an option that, instead of discarding the packet, signals the condition through an ECN-like mechanism combined with lower-priority forwarding of the packet. This informs the receiver directly about the problem and allows the application to deal with such conditions better than through other error signaling.

Data Model for gLBF packet metadata gLBF is specifically designed to support per-hop, per-flow stateless operation, because its processing requires no per-flow metadata, only per-packet metadata.

damper: 24-bit value, unit 1 usec This value is potentially different for every packet in a flow, and it is rewritten by every gLBF forwarder along the path. gLBF requires a packet header field indicating the delay that the damper on the next hop needs to apply to the packet. Dampening only needs to be as accurate as synchronous delivery needs to be. A unit of 1 usec is currently considered sufficient for all purposes. The maximum delay value is the maximum per-hop queuing latency. 65 msec is considered far more than will ever be needed. Therefore, a 16-bit header field is sufficient. Note that link propagation latency does not affect the delay value. Hence, the size of the delay field does not constrain the length of links.

end-to-end-priority: 3 bits This field needs to be the same for all packets of a flow, so that they are forwarded with the same latency processing and hence do not incur different latencies and possible reordering. This field is written when the packet enters the gLBF path and is only read afterwards. Eight priorities, and hence eight different latencies per hop, are considered sufficient. If the priority is an end-to-end packet header field, such as 8 different DSCPs in IP/IPv6, this results in 8 different latency traffic classes.

hop-by-hop-priority: 3 bits (per hop) This is a more advanced alternative to the end-to-end-priority. It is optional. If it is present for a particular hop, it supersedes the end-to-end-priority. Because this data is per-hop, it should not be encoded in an end-to-end part of the packet header, but in a hop-by-hop part of the header. Because the resources of each DetNet flow need to be controlled, rerouting DetNet flows without involving the controller-plane is highly undesirable and could easily result in congestion. A per-hop, per-flow stateless steering mechanism such as SR-MPLS or SRv6 is therefore highly desirable to provide a per-hop steering field. The priority could easily be made part of such a per-hop steering field by allocating, dynamically or on demand, up to 8 different SIDs (Segment IDentifiers), such as up to 8 MPLS labels, or by using 3 bits of the argument (ARG) field of SRv6 SIDs. When such a per-hop priority is indicated, the controller-plane can support many more than 8 end-to-end latencies simply by combining different per-hop latencies. For example, one flow may use priorities 1-1-1-1 across four hops, the next, slower flow 1-2-1-2, the next one 2-2-2-2, and so on.

phop-prio: 3 bits This field is rewritten on every hop when hop-by-hop priorities are used. It is an optional field that may be beneficial in an end-to-end header if hop-by-hop priorities are used for forwarding in gLBF and the hop-by-hop-priority of the packet on the prior hop is no longer available from the hop-by-hop packet header. This is typically the case in MPLS-based forwarding such as SR-MPLS, because this information will have been removed. Note that this header field is only required to support optional simplified high-speed forwarding implementations.

Error handling data items Di: one bit Downgrade intent. This bit is set when the packet enters the gLBF domain and is never changed. Ds: one bit Downgrade status. This bit is set to 0 when the packet enters the gLBF domain and may be changed to 1 by a gLBF forwarder, as described below. When the Di bit is not set and a gLBF forwarder recognizes that the packet cannot be forwarded within its guaranteed latency, the packet is discarded and forwarding-plane-specific error signaling (such as ICMP/ICMPv6) is triggered. Discarding is done to avoid causing latency errors further down the path because of this packet. When the Di bit is set, the packet is not discarded; instead, the Ds bit is set to indicate that the packet must no longer be forwarded with gLBF guaranteed latency, but only as best-effort or even lower-than-best-effort traffic. Di and Ds may be encoded in the two ECN bits for IP/IPv6 data planes. The required behavior should be backward compatible with existing ECN implementations, in case the packet unexpectedly passes a router that does not process it as gLBF.

Accuracy and sizing of the damper field considerations This memo recommends that the damper field in packet headers be 24 bits or larger and represent the dampening time with a resolution of 1 nsec. The following text explains the reasoning for this recommendation. The accuracy needed for the damper value in the packet header, as well as for the internal calculations performed for gLBF, depends on a variety of factors. The most important factor is the accuracy of the bounded latency desired/required by the applications. Even though Ethernet is defined as an asynchronous medium, its clock accuracy is required to be +/- 100 ppm (parts per million). For a 1500-byte packet on 100 Mbps Ethernet, the serialization latency difference between the fastest and slowest clock is 24 nsec. If gLBF is also to be supported on such lower-speed networks with multiple hops, these errors may add up (?), and it is likely not possible to provide end-to-end latency accuracy much better than 1 usec without requiring more accurate transmission timing through mechanisms such as PTP, which gLBF attempts to avoid or minimize. For higher-speed links, however, such short-term latency variations become irrelevant. In WAN deployments, propagation latency is on the order of milliseconds, e.g., ca. 1 msec for 250 km of fiber. Serialization latency on 1 Gbps Ethernet is ca. 1 usec for a 128-byte packet. It is likely that a latency accuracy on the order of 1 usec is sufficient when the round-trip time is potentially 1000 times larger. Based on these two simple data points, this memo considers an end-to-end latency accuracy of 1 usec to be the target. To avoid introducing additive errors, the resolution of the damper value needs to be higher. This memo therefore proposes a resolution of 1 nsec for damper values, which is also the resolution used in PTP. The maximum value, and hence the size, of the damper field depends on the maximum buffering latency in the sending node plus smaller factors such as serialization latency. This maximum depends primarily on the slowest links to be supported. A 128-byte packet on a 100 Mbps link has a serialization latency of ca. 10 usec. With a 16-bit damper value at nsec resolution, this would allow buffering of only 6 such packets, which may be too low. Hence, the damper field should be at least 24 bits.

Ingress and Egress processing

Network ingress edge policing When packets enter a gLBF domain from a prior hop, such as a node from a different domain or a sending host, and that prior hop sends gLBF packet markings, then the packet arrival timing and the gLBF markings in the packet header(s) MUST only be honored when that prior hop is trusted by the gLBF domain. If the prior hop cannot be trusted to mark and/or time packets correctly, the first-hop gLBF node in the gLBF domain MUST implement a per-flow policer to prevent packets from such a prior hop from compromising gLBF guarantees in the gLBF domain. The policer is logically placed after the gLBF damper module and before the queuing module. Instead of a policer, the ingress edge node of the gLBF domain MAY use a per-flow shaper or interleaved regulator. When, for example, a prior hop sends packets of a flow at too high a rate, a shaper delays them further and avoids discarding packets of that flow until the flow's burst size is also exceeded. A policer would immediately discard any packet that does not conform to its flow's envelope.

gLBF sender types

Simple gLBF senders Senders are expected to send their traffic flows such that their relative timing complies with the admitted traffic envelope. Senders that send only one flow, such as simple sensors or actuators, will typically be able to do this. When senders can do this (send gLBF packets with the right timing), these packets do not need gLBF packet header elements. Instead, the ingress network node can calculate the damper time locally, as described next, avoiding any need for gLBF processing in the sender. In this case, the maximum gLBF damper time is the serialization time of the largest gLBF packet received from the sender. Thus, when the sender sends packets smaller than the maximum admitted packet size, the first-hop network node needs to dampen packets by the difference in serialization time.

Normal gLBF senders When senders cannot fully comply with sending packets according to their admitted envelope, they need to implement gLBF and indicate the gLBF damper value in the packet header. For example, when the sender cannot ensure that the outgoing interface is free at the target sending time of a gLBF packet, another packet still being serialized will impact the timing of the following gLBF packet(s). Another example is a sender that has to send packets from multiple flows that cannot be, or are not, generated in a coordinated fashion; these packets can then compete for the sender's outgoing interface, introducing delay. Normal gLBF senders simply need to implement the queuing and marking stages of gLBF, but not the dampening stage, because that only applies to receivers/forwarders.

Non-gLBF senders When senders can neither act as simple gLBF senders nor implement the gLBF queuing and marking stages, the first-hop node of the gLBF domain needs to treat them as an untrusted prior hop, but always use a shaper or interleaved regulator so as not to discard their packets.

Receivers and gLBF

Normal gLBF receivers Normal gLBF receivers process gLBF packet markings and therefore need to implement the gLBF dampening stage.

gLBF-incapable receivers When receivers cannot implement the gLBF dampening stage, the worst-case burst that may arrive at the receiver has to be counted as added jitter of the end-to-end service. The network cannot eliminate such bursts without additional coordination and gating of packets across all senders and their network ingress nodes. This is not a gLBF-specific issue, but simply a limitation of the (near-)synchronous service model offered by gLBF, and it equally exists in any delay and latency model with this type of service: Consider a set of N senders that conspire to generate a burst at the same receiver. They know from the admission control model the latency each of them has to that receiver, and can thus time the generation of packets such that the last-hop router would have to send all N packets (one from each sender) on the interface simultaneously, which is obviously not possible. With gLBF-capable receivers, this possible burst is accounted for by the controller-plane including the latency introduced by such a worst-case burst in the end-to-end latency, and by indicating to the receiver, in the gLBF header field, the actual time the packet needs to be dampened. When the receiver does not support such dampening, however, that maximum last-hop burst simply turns into possible jitter.

Further considerations A host may be a different type of gLBF sender than gLBF receiver. A common case in cloud applications is a container or VM in a data center acting as sender/receiver. Such an application may be considered trusted by the gLBF domain if it belongs to the network operator itself (such as part of a higher-level service of the network operator), but not if it is a customer application. gLBF marking needs to happen after contention between packets from all applications, so it would typically happen inside the operating system. The operating system of such a host may not (yet) implement gLBF, so it may not be possible to correctly generate the gLBF marking as a sender. Instead, the application may generate the appropriate packet markings for steering and gLBF, but may have to leave the damper marking inaccurate. Conversely, dampening of received gLBF packets on a receiver can happen at the application level, so the operating system does not need to support it. Such dampening is the same functionality that is normally called "playout buffering" in applications, except that the delay is much smaller, because playout buffering needs to take the whole path delay into account, whereas gLBF dampening covers only the prior hop.

Summary There is a non-trivial set of ingress/egress processing options that could help to simplify and secure the handling of different types of senders and receivers. Later versions of this memo may define specific profiles of edge behavior to limit the recommended set of implementation options for senders/receivers and gLBF edge nodes.

Controller-plane considerations (informative)

gLBF versus UBS / TSN-ATS By relying on for both the traffic model and the bounded latency calculus, gLBF should be easy to operationalize using existing controller-plane implementations for TSN-ATS: UBS/TSN-ATS and gLBF provide the same bounded latency and use the same model to manage bandwidth for different flows, namely calculating which flow goes into which priority queue on which hop. The main change to a UBS/TSN-ATS controller-plane when using gLBF concerns what happens when a flow is admitted into the network, removed, or rerouted. In UBS, each of these operations requires the controller-plane to signal the traffic flow parameters to each node along the path, so that the forwarding plane can establish the per-hop, per-flow state for the flow. Depending on the network configuration, this also implies configuring the next hop of the flow for routing. For gLBF hop-by-hop operations, the first hop needs to receive the per-path or per-hop priority information, which it then imposes in an appropriate packet header of the flow's packets.

First-hop policing In gLBF, the controller-plane has to perform this action only towards the ingress node of the gLBF domain. Traffic parameters such as rate and burst size are only needed there to establish a policer, so that the flow cannot violate its admitted traffic parameters. This first-hop policing is actually a function independent of gLBF, UBS or any other method used along the path.

Path steering gLBF operates independently of the path steering mechanism, but the controller-plane will very likely want to ensure that gLBF (or, for that matter, any DetNet) traffic does not unexpectedly get rerouted by in-network routing mechanisms, because it normally will not, or cannot, reserve resources for such rerouted flows on arbitrary failure paths without significant additional effort and/or waste of resources.

Decentralized controller-plane Today's (pre-DetNet) controller-plane deployments commonly use one centralized "SDN" controller to implement the functions of the "Path Computation Element" (PCE), which tracks free bandwidth on every hop in the network topology and can hence calculate on which path enough resources are available for a requested new flow. This is, for example, the typical deployment with RSVP-TE and Segment Routing. However, it is equally possible to design a fully decentralized controller-plane, especially for DetNet networks that may not have all the complex policy requirements that make a centralized PCE most desirable. This is possible because, with gLBF, the controller-plane does not need to interact with non-ingress routers when dynamically allocating resources to new traffic flows. The model for a fully decentralized controller-plane could be derived from the model developed and deployed for RSVP-TE without a centralized PCE, where the ingress routers (LSRs) perform path calculations based on IGP-flooded information about the available resources on every hop. However, the necessary protocol operations may not lend themselves well to existing IGPs, and a new decentralized controller-plane may even need to be developed. Note, though, that none of these (complex) controller-plane considerations need to apply to deployments where the total amount of gLBF traffic is well defined and so small compared to the available capacity that no bandwidth exhaustion would occur, even under supported failure conditions. In those cases, gLBF is still necessary to guarantee the latency of the traffic, but the whole controller-plane is reduced to a network capacity planning and ingress-router configuration task. This "controller-plane-free" mode of operation is standard practice in many large service provider networks for traffic that requires guaranteed bandwidth but has statically known (limited) capacity requirements, such as IPTV.

IANA considerations None yet.

Acknowledgements Thanks to Florian Kauer for his input.

Changelog [RFC-editor: please remove] Individual document (draft-eckert-detnet-glbf) 00 Initial version. 01 Added use cases from Stefan 02 refresh only, waiting for progress in WG discussion 03 refresh, affiliation change of co-author 04 refresh, still waiting for more discussion in detnet 05 refresh, still waiting for more discussion in detnet 06 added Solution Scope, Taxonomy and Status for DetNet adoption considerations 07 Fixed author address (hommes) WG document (draft-ietf-detnet-glbf) 00 Initial version 01 Just refreshing. Ran out of time to incorporate outstanding feedback for IETF126. Sorry. Will get to it within a month. 02 Updated author address (hommes). Changes in response to the WG adoption review by Florian Kauer: Summary: added that with gLBF the controller-plane only needs to interact with (and per-flow state is only needed on) ingress routers, with pointers to the controller-plane considerations. New section "decentralized controller plane": centralized SDN/PCE controller-plane as today's deployment model, option for a decentralized controller-plane, and "controller-plane-free" deployments. Fixed "TSN-ATN" -> "TSN-ATS" and "not starting" -> "now starting". Resolved TBDs: clock synchronization cost factor, MAX1 fine-tuning. Other changes: Summary: per-hop fixed latency guarantees, end-to-end latency as sum of per-hop latencies. Clock synchronization challenges: stated that gLBF reduces clock synchronization requirements. Status: added the declarative traffic engineering implementation (). Added Acknowledgements section. 03 Editorial English fixes throughout (grammar, wording, typos, punctuation, US spelling, hyphenation). Semantic fixes: Damper principle: delay is (M-L), not (B-L). Damper with UBS: gLBF provides on-time (not in-time) latency service. Link propagation latency P is medium length divided by speed of light (not times). High speed appendix: PIFO (not FIFO) hardware implementations are still research. PCE is "Path Computation Element". Terminology: TCQF is no longer called "the mechanism specified in this memo". Taxonomy: TCQF replaced by gLBF (copy-paste error). Fixed titles of the UBS strict priority queuing figure and the glbf_send() figure. Per-hop priority example: 1-1-1-1 across four hops (was five). IGMP -> ICMP, PPT -> PTP. Damper field accuracy: serialization (not propagation) latency in the 24 nsec and 10 usec examples.

This note describes the use of the RSVP resource reservation protocol with the Controlled-Load and Guaranteed QoS control services. [STANDARDS-TRACK] This memo describes the network element behavior required to deliver a guaranteed service (guaranteed delay and bandwidth) in the Internet. [STANDARDS-TRACK] This document defines the IP header field, called the DS (for differentiated services) field. [STANDARDS-TRACK] This document defines a flexible solution for support of Differentiated Services (Diff-Serv) over Multi-Protocol Label Switching (MPLS) networks. [STANDARDS-TRACK] This document provides the overall architecture for Deterministic Networking (DetNet), which provides a capability to carry specified unicast or multicast data flows for real-time applications with extremely low data loss rates and bounded latency within a network domain. Techniques used include 1) reserving data-plane resources for individual (or aggregated) DetNet flows in some or all of the intermediate nodes along the path of the flow, 2) providing explicit routes for DetNet flows that do not immediately change with the network topology, and 3) distributing data from DetNet flow packets over time and/or space to ensure delivery of each packet's data in spite of the loss of a path. DetNet operates at the IP layer and delivers service over lower-layer technologies such as MPLS and Time- Sensitive Networking (TSN) as defined by IEEE 802.1. This document specifies version 6 of the Internet Protocol (IPv6). It obsoletes RFC 2460. This document specifies the Deterministic Networking (DetNet) data plane when operating over an MPLS Packet Switched Network. It leverages existing pseudowire (PW) encapsulations and MPLS Traffic Engineering (MPLS-TE) encapsulations and mechanisms. This document builds on the DetNet architecture and data plane framework. This document describes the use of RSVP (Resource Reservation Protocol), including all the necessary extensions, to establish label-switched paths (LSPs) in MPLS (Multi-Protocol Label Switching). Since the flow along an LSP is completely identified by the label applied at the ingress node of the path, these paths may be treated as tunnels. A key application of LSP tunnels is traffic engineering with MPLS as specified in RFC 2702. [STANDARDS-TRACK] This document describes extensions to Resource Reservation Protocol - Traffic Engineering (RSVP-TE) for the set up of Traffic Engineered (TE) point-to-multipoint (P2MP) Label Switched Paths (LSPs) in Multi- Protocol Label Switching (MPLS) and Generalized MPLS (GMPLS) networks. The solution relies on RSVP-TE without requiring a multicast routing protocol in the Service Provider core. Protocol elements and procedures for this solution are described. There can be various applications for P2MP TE LSPs such as IP multicast. Specification of how such applications will use a P2MP TE LSP is outside the scope of this document. [STANDARDS-TRACK] Bit Index Explicit Replication (BIER) is an architecture that provides optimal multicast forwarding through a "multicast domain", without requiring intermediate routers to maintain any per-flow state or to engage in an explicit tree-building protocol. When a multicast data packet enters the domain, the ingress router determines the set of egress routers to which the packet needs to be sent. The ingress router then encapsulates the packet in a BIER header. The BIER header contains a bit string in which each bit represents exactly one egress router in the domain; to forward the packet to a given set of egress routers, the bits corresponding to those routers are set in the BIER header. The details of the encapsulation depend on the type of network used to realize the multicast domain. This document specifies a BIER encapsulation that can be used in an MPLS network or, with slight differences, in a non-MPLS network. Segment Routing (SR) leverages the source routing paradigm. A node steers a packet through an ordered list of instructions, called "segments". A segment can represent any instruction, topological or service based. A segment can have a semantic local to an SR node or global within an SR domain. SR provides a mechanism that allows a flow to be restricted to a specific topological path, while maintaining per-flow state only at the ingress node(s) to the SR domain. SR can be directly applied to the MPLS architecture with no change to the forwarding plane. A segment is encoded as an MPLS label. An ordered list of segments is encoded as a stack of labels. The segment to process is on the top of the stack. Upon completion of a segment, the related label is popped from the stack. SR can be applied to the IPv6 architecture, with a new type of routing header. A segment is encoded as an IPv6 address. An ordered list of segments is encoded as an ordered list of IPv6 addresses in the routing header. The active segment is indicated by the Destination Address (DA) of the packet. The next active segment is indicated by a pointer in the new routing header. Segment Routing can be applied to the IPv6 data plane using a new type of Routing Extension Header called the Segment Routing Header (SRH). This document describes the SRH and how it is used by nodes that are Segment Routing (SR) capable. This document provides an overall framework for the Deterministic Networking (DetNet) data plane. It covers concepts and considerations that are generally common to any DetNet data plane specification. It describes related Controller Plane considerations as well. The Segment Routing over IPv6 (SRv6) Network Programming framework enables a network operator or an application to specify a packet processing program by encoding a sequence of instructions in the IPv6 packet header. Each instruction is implemented on one or several nodes in the network and identified by an SRv6 Segment Identifier in the packet. This document defines the SRv6 Network Programming concept and specifies the base set of SRv6 behaviors that enables the creation of interoperable overlays with underlay optimization. This document describes the flow and service information model for Deterministic Networking (DetNet). These models are defined for IP and MPLS DetNet data planes. This memo describes per-packet stateless strict and loose path steered replication and forwarding for "Bit Index Explicit Replication" (BIER) packets (RFC 8279); it is called "Tree Engineering for Bit Index Explicit Replication" (BIER-TE) and is intended to be used as the path steering mechanism for Traffic Engineering with BIER. BIER-TE introduces a new semantic for "bit positions" (BPs). These BPs indicate adjacencies of the network topology, as opposed to (non-TE) BIER in which BPs indicate "Bit-Forwarding Egress Routers" (BFERs). A BIER-TE "packets BitString" therefore indicates the edges of the (loop-free) tree across which the packets are forwarded by BIER-TE. BIER-TE can leverage BIER forwarding engines with little changes. Co-existence of BIER and BIER-TE forwarding in the same domain is possible -- for example, by using separate BIER "subdomains" (SDs). Except for the optional routed adjacencies, BIER-TE does not require a BIER routing underlay and can therefore operate without depending on a routing protocol such as the "Interior Gateway Protocol" (IGP). This document presents a timing model for sources, destinations, and Deterministic Networking (DetNet) transit nodes. Using the model, it provides a methodology to compute end-to-end latency and backlog bounds for various queuing methods. The methodology can be used by the management and control planes and by resource reservation algorithms to provide bounded latency and zero congestion loss for the DetNet service. Sangmyung University Huawei ZTE Corporation Futurewei Technologies This draft is to facilitate the understanding of the data plane enhancement solutions, which are suggested currently or can be suggested in the future, for deterministic networking. This draft provides criteria for classifying data plane solutions. Examples of each category are listed, along with reasons where necessary. Strengths and limitations of the categories are described. Suitability of the solutions for various services of deterministic networking are also mentioned. Reference topologies for evaluation of the solutions are given as well. Futurewei Technologies USA This memo explain requirements, benefits and feasibility of a new DetNet service function tentatively called "flow interleaving". It proposes to introduce this service function into the DetNet architecture. Flow interleaving can be understood as a DetNet equivalent of the IEEE TSN timed gates. Its primary role is intended to be at the ingress edge of DetNet domains supporting higher utilization and lower bounded latency for flow aggregation (interleaving of flows into a single flow), as well as higher utilization and lower bounded latency for interleaving occurring in transit hops of the DetNet domain in conjunction with in-time per-hop bounded latency forwarding mechanisms. Huawei Huawei China Mobile Sangmyung University ETRI One of the goals of DetNet is to provide bounded end-to-end latency for critical flows. This document defines how to leverage Segment Routing (SR) to implement bounded latency. Specifically, the SR Identifier (SID) is used to specify transmission time (cycles) of a packet. When forwarding devices along the path follow the instructions carried in the packet, the bounded latency is achieved. This is called Cycle Specified Queuing and Forwarding (CSQF) in this document. Since SR is a source routing technology, no per-flow state is maintained at intermediate and egress nodes, SR-based CSQF naturally supports flow aggregation that is deemed to be a key capability to allow DetNet to scale to large networks. Huawei Huawei Huawei Huawei China Mobile This document presents the overall framework and key method for Large-scale Deterministic Network (LDN). LDN can provide bounded latency and delay variation (jitter) without requiring precise time synchronization among nodes, or per-flow state in transit nodes. Futurewei Technologies USA Huawei Technologies University of Surrey ICS Malis Consulting ETRI China Mobile Huawei Technologies Huawei Technologies This memo specifies a forwarding method for bounded latency and bounded jitter for Deterministic Networks and is a variant of the IEEE TSN Cyclic Queuing and Forwarding (CQF) method. Tagged CQF (TCQF) supports more than 2 cycles and indicates the cycle number via an existing or new packet header field called the tag to replace the cycle mapping in CQF which is based purely on synchronized reception clock. This memo standardizes TCQF as a mechanism independent of the tagging method used. It also specifies tagging via the (1) the existing MPLS packet Traffic Class (TC) field for MPLS packets, (2) the IP/IPv6 DSCP field for IP/IPv6 packets, and (3) a new TCQF Option header for IPv6 packets. Target benefits of TCQF include low end-to-end jitter, ease of high- speed hardware implementation, optional ability to support large number of flow in large networks via DiffServ style aggregation by applying TCQF to the DetNet aggregate instead of each DetNet flow individually, and support of wide-area DetNet networks with arbitrary link latencies and latency variations as well as low accuracy clock synchronization. Huawei Huawei This document presents a queuing mechanism with multiple cyclic buffers. Futurewei Technologies Inc. University of Tuebingen KOEVOO This memo describes per-packet stateless strict and loose path steered replication and forwarding for "Bit Index Explicit Replication" (BIER) packets (RFC 8279); it is called "Tree Engineering for Bit Index Explicit Replication" (BIER-TE) and is intended to be used as the path steering mechanism for Traffic Engineering with BIER. BIER-TE introduces a new semantic for "bit positions" (BPs). These BPs indicate adjacencies of the network topology, as opposed to (non-TE) BIER in which BPs indicate "Bit-Forwarding Egress Routers" (BFERs). A BIER-TE "packets BitString" therefore indicates the edges of the (loop-free) tree across which the packets are forwarded by BIER-TE. BIER-TE can leverage BIER forwarding engines with little changes. Co-existence of BIER and BIER-TE forwarding in the same domain is possible -- for example, by using separate BIER "subdomains" (SDs). Except for the optional routed adjacencies, BIER-TE does not require a BIER routing underlay and can therefore operate without depending on a routing protocol such as the "Interior Gateway Protocol" (IGP). Huawei Technologies Co. Ltd EPFL EPFL Huawei Technologies Co. Ltd Ericsson This document presents a timing model for sources, destinations, and Deterministic Networking (DetNet) transit nodes. Using the model, it provides a methodology to compute end-to-end latency and backlog bounds for various queuing methods. The methodology can be used by the management and control planes and by resource reservation algorithms to provide bounded latency and zero congestion loss for the DetNet service. RAD Routers perform two distinct user-plane functionalities, namely forwarding (where the packet should be sent) and scheduling (when the packet should be sent). One forwarding paradigm is segment routing, in which forwarding instructions are encoded in the packet in a stack data structure, rather than programmed into the routers. Time Sensitive Networking and Deterministic Networking provide several mechanisms for scheduling under the assumption that routers are time synchronized. The most effective mechanisms for delay minimization involve per-flow resource allocation. SRTSN is a unified approach to forwarding and scheduling that uses a single stack data structure. Each stack entry consists of a forwarding portion (e.g., IP addresses or suffixes) and a scheduling portion (deadline by which the packet must exit the router). SRTSN thus fully implements network programming for time sensitive flows, by prescribing to each router both to-where and by-when each packet should be sent. Futurewei Technologies USA Stewart Bryant Ltd The purpose of this memo is to explain the challenges and limitations of existing (standardized) bounded latency queuing mechanisms for desirable (large scale) MPLS and/or IP based networks to allow them to support DetNet services. These challenges relate to low-cost, high-speed hardware implementations, desirable network design approaches, system complexity, reliability, scalability, cost of signaling, performance and jitter experience for the DetNet applications. Many of these problems are rooted in the use of per- hop, per-flow (DetNet) forwarding and queuing state, but highly accurate network wide time synchronization can be another challenge for some networks. This memo does not intend to propose a specific queuing solution, but in the same way in which it describes the challenges of mechanisms, it reviews how those problem are addressed by currently proposed new queuing mechanisms. IEEE 802.1 Working Group IEEE Time-Sensitive Networking (TSN) Task Group. IEEE Time-Sensitive Networking (TSN) Task Group. Future Generation Computer Systems, Vol. 182, 108494, ISSN 0167-739X

High speed implementation considerations

High speed example pseudocode instead of reference pseudocode This memo does not provide normative reference code aligned with the normative functional description in . The primary reason is that a naive 1:1 implementation of the functional description would lead to inferior performance. Nevertheless, the feasibility of high-speed implementations is an important factor in the deployability of mechanisms like gLBF. Therefore, this appendix describes considerations for such high-speed implementations, including pseudocode for one possible option. describes two possible optimized hardware implementation approaches for gLBF: one using "Push In First Out" (PIFO) queues, the other using timed FIFO queues. The following describes and explains the second approach, because it is hopefully more feasible for nearer-term hardware implementations, given that scalable high-speed PIFO hardware implementations are still a matter of research. However, the timed FIFO algorithm presented here may also face hardware scalability challenges, because it requires O(IIF * priorities^2) FIFO queues per outgoing interface, where IIF is the number of (incoming) interfaces on the node and priorities is the number of desired priority levels (TSN allows up to 8). If, however, the priority of flows (packets) is not per-hop but the same on every hop (per-path), then the number is just (IIF * priorities).

UBS high speed implementations The algorithm for the pseudocode shown later in this appendix is based on the analysis of gLBF behavior performed in . That analysis reapplies the same type of analysis and algorithm as , which can be used to implement TSN-ATS in high-speed switching hardware. Therefore, this appendix starts by explaining the UBS mechanisms. UBS (see , Figure 4) logically consists of two separate stages. The first stage is a per-priority, per-incoming-interface (IIF) interleaved regulator. An interleaved regulator is a FIFO where dequeuing of the queue head packet (qhead) is based on the per-flow state of qhead. A target departure time for the qhead is calculated from the flow state that the router maintains across packets, and once that departure time is reached, the packet is passed to one of the egress strict priority queues. It is called an interleaved regulator because the FIFO holds packets from multiple flows (all from the same incoming interface and with the same outgoing priority), hence 'interleaved', and 'regulator' because it delays the packet until the target departure time. The innovation of interleaved regulators, compared to the earlier per-flow shapers as used in , is the recognition and mathematical proof that the arrival and target departure times of all packets received from the same IIF (prior-hop node) with the same egress priority are in the same order, so that these packets can be enqueued into a single FIFO in which per-flow processing only needs to happen for the qhead, with no more than negligible added latency compared to per-flow shapers. With this understanding, the interleaved regulator and the following strict priority stage can easily be combined into a single queuing stage with appropriate scheduling. In this "flattened" approach, each logical egress strict priority queue consists of the per-IIF interleaved regulators (FIFO queues) for that priority. Egress scheduling now needs to perform the per-flow state processing for the qhead of each queue, and then take the target departure time of that packet into account, selecting the interleaved regulator whose qhead has the highest priority and earliest target departure time.

gLBF compared to UBS The same line of thought as in UBS can be applied to gLBF and is the basis for the algorithm and pseudocode described here. In gLBF, the order of packets by arrival and target departure time depends on the egress priority and the IIF, but also on the priority the packet had on the prior hop (pprio). All packets with the same (IIF,prio,pprio) arrive in the same order in which they need to depart, so it is possible to enqueue them into a FIFO and only perform timed dequeuing of the qhead. Both prio and pprio matter because, when gLBF uses an encapsulation that allows per-hop priority indications, a packet's priority can differ on each hop. In UBS, the priority of a packet can equally differ across hops, but the prior-hop priority is not needed to separate packets into interleaved regulators, because UBS targets in-time delay, where the interleaved regulator only needs to compensate for burst accumulation. In gLBF, in contrast, the delay to be introduced depends on the damper value, which depends on the prior-hop priority. As a result, the order in which gLBF packets should depart (absent any burst accumulation) also depends on the prior-hop priority. Because gLBF does not deal with per-flow state on the router as UBS does in its interleaved regulators, the FIFOs for gLBF are called timed FIFOs in this memo rather than interleaved regulators.

Comparison to normative behavior Whereas the normative description defines D, Q and M blocks, where the D block performs the dampening and the Q block performs the priority queuing, the high-speed hardware optimization replaces D and Q with a single DQ block. In it, packets are enqueued into a single-stage per-(IIF,pprio,prio) FIFO (similar to UBS), and the dampening by the damper time happens on dequeuing from that stage (also similar to UBS), while packets are scheduled/prioritized based on their prio.

IIF| +-+ +-------+ +-+ |OIF IIF| +-+ +-------+ +-+ |OIF +------------------------+ +------------------------+ |<- Dampened Q ->| |<- Dampened Q ->| |<- A/B in-time latency ->| |<--A/B on-time latency -------->| ]]>

In the normative definition, the damper value to be put into the packet depends on the time y at which the packet leaves the damper stage D and enters the priority queue stage Q. Simplified, the damper value is (MAX1 - (z - y)). In the high-speed implementation, packets are not passed from a damper stage queue to a priority queue; instead, they stay in the same queue. On the one hand, this is one of the core reasons why this approach can support high speed: it does not require two-stage enqueuing/dequeuing. On the other hand, it also eliminates the ability to take a timestamp at time y. To replicate the normative gLBF behavior with a single stage, the marking stage M (where the new damper value is calculated) therefore needs to know y by other means. Fortunately, y is exactly the target departure time of the packet from the D stage, and therefore also from the DQ stage, so it already needs to be calculated on entry to DQ: y = (x + damper - <details>), as explained below. The pseudocode consists of glbf_enqueue() describing the enqueuing into the DQ stage, and glbf_dequeue() describing the dequeuing from the DQ stage. glbf_dequeue() (synchronously) calls glbf_send() which represents the marking stage M.

Pseudocode

glbf_enqueue()

pak.header contains the parsed fields of the received gLBF packet; tdamp is the damper value. pak.context is a node-local header of the packet. iif is the interface on which the packet was received. prio and pprio are the current and prior-hop priorities from the packet header. If, for example, SRH () is used with gLBF, and SIDs are formatted according to , then the priority for each hop could be expressed as a 4-bit SID ARG field (see , Section 3.1), indicating 16 different priorities per hop. Parsing the prior-hop priority is not a normative requirement of gLBF, but a specific requirement of the high-speed algorithm presented here, which allows the use of single-stage timed FIFOs instead of PIFOs. This type of parsing is not required by SRH, but it is a benefit of packet headers that, like SRH, retain the prior hops' information, as opposed to the SR-MPLS approach, where past-hop SIDs/labels are discarded. now() takes a timestamp from a local (unsynchronized) clock at the time of execution. [1] pak.context.ta (time of arrival) is the calculated time at which the first bit of the packet entered the incoming interface of the node. adj_rcv_time() is defined in the next subsection. td is the target departure time, calculated from the current time and the damper value from the packet. It simply converts the relative damper value into an absolute timestamp. max1[oif,prio] is the only gLBF-specific interface-level state maintained (read-only) across packets. It is the maximum time calculated by admission control in the controller-plane for each priority. It is copied into a context header field here, even though it is only used later during dequeuing, on the assumption that accessing state information is undesirable in the time-critical dequeuing/serialization stage, whereas such state lookups are very common for many forwarding plane features before enqueuing. Finally, the packet is enqueued into a per-(oif,iif,prio,pprio) timed FIFO queue.

adj_rcv_time() As the reference time against which the damper value in the packet is applied, this specification assumes the time when the first bit of the packet is seen on the medium connected to the incoming interface.

In most simple node equipment, the primary variable latency between that (first-bit) measurement point and the time at which the now parameter of adj_rcv_time() can be taken is the deserialization time of the packet. It can easily be calculated from the packet length, assumed to be known from pak.context.length, and the bit rate of the incoming interface, known from pak.context.iif.speed, plus a fixed, measured or calculated latency FUDGE for other internal processing. In the example pseudocode, these are assumed to be static. If other internal factors in prior forwarding within the node create significant variation, they may need to be compensated through additional mechanisms. In this case, rather than as shown in glbf_enqueue(), the reference time should be taken directly upon receipt of the packet, before it enters the forwarding stage F, and then be used as the now parameter of adj_rcv_time(). This is deliberately not shown in the code, to highlight the simplest implementation option, which should be sufficient for most node types.

glbf_dequeue()

glbf_dequeue() selects as the gLBF packet to send the enqueued packet with the highest priority and the earliest departure time td (as calculated in glbf_enqueue()), where td must also not be in the future, because otherwise the packet still needs to be dampened. The outer loop across this hop's prio finds the packet with exactly these properties, dequeues it and sends it. Strict priority is expressed through the term maxpriority...minpriority, which searches from maximum to minimum priority. The two inner loops simply look for the packet with the earliest target departure time across all the (IIF,pprio) timed FIFO queues for the same prio and OIF. In ASIC/FPGA implementations, finding this packet is not a sequential operation as shown in the pseudocode, but can be a single-clock-cycle parallel compare operation across the td values of all queue qheads, at the expense of chip area, similar to how TCAMs work.

glbf_send()

glbf_send() represents the marking block M of the specification. It needs to calculate the new damper value td and update it in the packet header before sending the packet. td needs to be the maximum time max1 that the packet could have stayed in priority queuing, minus the time qtime that it actually stayed in the queue, minus any additional time FUDGE2 that the packet still needs to be processed by the router before its first bit appears on the sending interface. This is calculated as shown in [1] and [2] of . The primary factor impacting FUDGE2 is whether this calculation and the update of the td field in the packet header can be done directly before serialization starts, or whether they may need to be done while another packet is still being serialized on the interface. Taking the added latency of a packet currently being serialized into account can, for example, be implemented by remembering the time when that packet started to be serialized and its length, and then using these to calculate FUDGE2.

Performance considerations PTP clock synchronization needs to capture the accurate arrival time of PTP packets. It also measures the residence time of PTP packets between receiving and sending them with nsec accuracy, and adds it to a PTP packet header field (correctionField). Given that this needs to be hardware-supported functionality, it is unclear whether there is a relevant difference between supporting it for a few PTP signaling packets and for a much larger number of gLBF data packets, or whether it is a good indication of the feasibility of the packet processing steps that gLBF requires.

History and comparison with other bounded latency methods Before this work, the best solutions for the requirements outlined in this document were and , which have been proven feasible for high-speed forwarding planes through vendor high-speed forwarding plane implementations using low-cost FPGAs for cyclic queuing. On the other hand, implementation analysis concluded that was infeasible for the same type of high-speed, low-cost hardware, due to the need for high-speed flow-state operations, especially read/write cycles to update the state for every packet at packet processing speeds. While and are very good solutions, proven to work at high speed and low cost, available for deployment and ready for standardization, the need for network-wide, hop-by-hop clock synchronization (albeit at lower accuracy than required by other mechanisms feasible at high speed and low cost), as well as the need to find good network-wide compromise cycle times, still makes planning and managing them in dynamic, large-scale and/or low-cost networks more complex to operationalize than the authors would like. In parallel to the short-term operationalizable solutions and , was researched, exploring a wide range of options to signal and control latency through advanced per-hop processing and new latency-related in-packet header elements. Unfortunately, given the flexibility of , it was impossible to find a calculus for simple admission control. Ultimately, was designed out of the desire for a mechanism derived from that also provides guaranteed bounded latency and has the flexibility benefits of with respect to fine-grained, per-hop independent latency management, but hopefully still fits within the limits of what can be implemented in high-speed, low-cost forwarding/queuing hardware. The feasibility of gLBF in high-speed forwarding hardware has yet to be proven. The following considerations explain why the authors think it should be quite feasible: The total number of FIFOs a platform needs to support is comparable to the number of queues already supported in the same type of devices today, except that not all service provider core nodes may implement that many, because in the absence of DetNet services there is no requirement for them. Implementing timed FIFOs (if the proposed high-speed implementation approach is used) is no more complex than implementing shapers, which have by now also started to appear on high-speed (100 Gbps and faster) core routers, primarily for high-speed virtual leased line services. For the rewriting of packet header fields based on the calculated latency the packet experienced in the node, IOAM-type functionality arguably already provides evidence of feasibility in high-speed hardware forwarding planes. Processing of PTP timing protocol packets has similar requirements. Even if implementation challenges at high speed and low cost make gLBF a longer-term option in those networks, low-speed (1/10 Gbps) implementations, such as in manufacturing or cars, may be an interesting option to explore, given that gLBF has the no-jitter benefits of without the need for cycle management or clock synchronization: the best of both worlds, and TSN-ATS.

Solution Scope, Taxonomy and Status for DetNet adoption considerations

Solution scope gLBF is not proposed as a standalone solution, but in conjunction with , a per-flow mechanism used on the ingress gLBF router to allow more flexible per-flow management of the admission of flow bursts than presented in this document. The proposal is split into two documents because can be combined with all "mostly synchronous" per-hop mechanisms, for example, besides gLBF, also CQF and .

Taxonomy Using the section numbers/titles of the taxonomy document , gLBF can be classified as follows. In summary, gLBF is very much like TCQF, except that it replaces cycles with Dampers, thereby further reducing end-to-end jitter, reducing clock accuracy requirements (only phase synchronization, not clock synchronization, is required) and removing the complexity of end-to-end cycle management. Compared to older Damper-based proposals, gLBF reuses the ATS calculus, which allows reuse of ATS admission control and delay calculations, and a simple implementation based on FIFOs with timed scheduling (like ATS). 4.1 Per Hop Dominant Factor for Latency Bound The per-hop dominant latency bound is Category 3. The per-hop latency for each class is determined by the configured burst size. 5.1 Periodicity gLBF is not periodic. Its fundamental mode of operation is not based on cycles to manage bursts, but on per-hop, per-packet synchronous delivery, which maintains burst sizes end-to-end across all hops. 5.2. Traffic Granularity Formally, gLBF has class-level granularity. However, the statement in (revision 04) that all packets within a class are treated the same is questionable for gLBF (and for Damper solutions in general): in Damper solutions like gLBF, packet latency is calculated per packet (within the same class) based on the previously experienced per-hop latency, in order to compensate for it. 5.3. Time Bounds gLBF is per-hop bounded. It is Right Bounded by the configured burst size for the hop and class of the packet. It is Left Bounded by the signaled remaining latency needed to achieve synchronous forwarding. If the bound were observed not on the wire, but at a specific forwarding step in the router (after reception, before enqueuing), then the per-hop bound would be extremely tight, limited only by implementation accuracy. 5.4 Service Order gLBF service order is solely arrival based. Suitable Categories for DetNet gLBF is "6.5. Class level non-periodic bounded category" according to the categorization.

Status gLBF primarily has an open-source, simulation-level validation implementation, as described in . Additional implementations were done in the Linux kernel by a university (TBD reference) and by a second university in conjunction with a Prolog-based admission control server implementation ().