Deterministic Networking (DetNet) Data Plane - guaranteed Latency Based Forwarding (gLBF) for bounded latency with low jitter and asynchronous forwarding in Deterministic Networks

1.1. Terminology

CaaS

Control as a Service. Cloud (compute) and network services that enable control (loops) for network-connected devices, for example cars.¶

CQF

Cyclic Queuing and Forwarding. A queuing mechanism defined in Annex T of IEEE 802.1Q.¶

DT

Dead Time. A term from CQF indicating the time during each cycle in which no frames can be sent because the receiving node could not receive them into the desired cycle buffer.¶

node

A system that, with respect to gLBF, does not act as a host (sender/receiver). This memo avoids the term router so as not to imply an IP/IPv6 router as opposed to an LSR (label switching router). Likewise, a node can also be an IEEE 802.1 bridge implementing gLBF.¶

TCQF

Tagged Cyclic Queuing and Forwarding. A mechanism proposed to DetNet, see [TCQF].¶

1.2. Summary

This memo proposes a mechanism called "guaranteed Latency Based Forwarding" (gLBF) for hop-by-hop packet forwarding with per-hop deterministically bounded latency and minimal jitter.¶

gLBF is intended to be useful across a wide range of networks and applications that need high-precision deterministic networking services, from in-car networks and networks for industrial automation on factory floors all the way to country-wide networks at 100 Gbps and beyond.¶

At its foundation, gLBF addresses the problems of burst accumulation and jitter accumulation across multiple hops to achieve per-hop fixed latency guarantees. This allows easy calculation of end-to-end latency as the sum of these per-hop latencies.¶

In particular, gLBF eliminates the need for the controller-plane to interact with the intermediate-hop routers of a gLBF path. This simplifies the controller-plane and network operations, and increases the number of flows the network can support: with gLBF, per-flow state is only required on ingress routers. Although the controller-plane in general is out of scope for this document, Section 4 discusses aspects relevant to system design with gLBF, including the opportunity for a decentralized controller-plane/admission control (Section 4.4).¶

Burst accumulation is the phenomenon whereby bursts of packets from senders in an admission-controlled network grow across intermediate nodes. It can only be managed with exponential complexity in admission control processing and a significant increase in worst-case end-to-end latency and/or lower maximum utilization. Dynamic, large-scale or easy-to-manage admission control solutions need forwarding mechanisms without this problem, so that admission control for bandwidth and jitter/buffer requirements can be linear: decomposed into purely per-hop calculations that are independent of changes in prior-hop traffic characteristics. This is not possible without forwarding-plane solutions that overcome burst accumulation.¶

Existing solutions address burst accumulation by maintaining inter-packet gaps on a per-flow basis, as in TSN Asynchronous Traffic Shaping (TSN-ATS). gLBF instead ensures that inter-packet gaps are maintained without the need for per-flow state. The basic idea is to assign a specific queuing delay budget to each node and traffic class. This budget is known in advance from the admission control calculus. When the packet is transmitted, the queuing delay it actually experienced in the node is subtracted from that budget, and the result is carried in a new header field of the packet. On receipt, the next node passes the packet through a delay stage, where the packet waits for the time indicated in that field before the node proceeds with regular processing. In this way, queuing delay variations are absorbed, and deterministic delay is achieved without the possibility of burst accumulation.¶

By addressing burst accumulation, gLBF also overcomes the problem of jitter accumulation, the second core problem of mechanisms such as TSN-ATS: depending on the amount of competing admitted traffic on a hop at any point in time, packets of a flow may experience anything from zero to the maximum delay across the hop. This is the per-hop jitter. It adds up across multiple hops, so applications requiring (near-)synchronous packet delivery cannot rely solely on such mechanisms. It likewise limits how highly networks with a large number of bounded latency flows can be utilized.¶

The basic principle on which gLBF operates was proposed by researchers in the early 1990s under the name "Dampers". Dampers were not pursued or adopted at the time because network equipment lacked the necessary capabilities. Only recent and upcoming improvements in advanced forwarding planes make it possible to build them at scale and at acceptable cost.¶

Unlike other proposals, gLBF does not require network-wide clock synchronization. Likewise, it does not need to maintain per-flow state on network nodes, because the delay budget and the queuing delay variations to be absorbed are carried in the packets themselves, making them "self-contained". This eliminates the per-flow, per-hop state maintenance required by TSN-ATS, with its scaling problems of signaling per-flow state from the controller-plane to every hop, and with the cost, in high-speed networking hardware, of the high-speed read/write memory accesses needed to retrieve, process and update these per-flow state variables for a large number of flows.¶

Unlike other damper proposals, gLBF also supports the queuing model and calculus of Urgency Based Scheduling (UBS, [UBS]), which is used by TSN-ATS. gLBF is intended as a plug-in replacement for, or a mechanism in parallel to, TSN-ATS, because it allows reuse of the same controller-plane design that selects paths for TSN-ATS, sizes TSN-ATS queues, calculates latencies and admits flows onto the calculated paths with the calculated latencies.¶

While gLBF as presented here is intended for use with IETF forwarding protocols, to provide DetNet QoS with bounded latency and lower, bounded jitter, it would be equally applicable to other forwarding planes, such as IEEE 802.1 Ethernet switching, assuming appropriate packet headers are defined to carry the hop-by-hop and end-to-end metadata required by the mechanism.¶

1.3. Application scenarios and use cases

gLBF addresses the same use cases that are targeted by deterministic networking and high-precision networking services in general. Common requirements of those services include the need to provide service levels within very tightly defined bounds, in particular very specific latencies without the possibility of congestion-induced loss. The ability to provide services with corresponding service level guarantees enables many applications that would simply not be feasible without such guarantees. The following describes some use cases and application scenarios.¶

The development towards autonomous vehicles leads to new requirements and a high demand for computational resources that are often not available within a car. One solution is to reduce in-car processing to a minimum and to offload computationally expensive tasks to the cloud. Because of the safety implications of these use cases, delays in the transport of messages from the cloud to the car and vice versa are often not acceptable. This includes application scenarios such as trajectory planning for driverless vehicles, but also emergency notifications to surrounding cars, which require fast delivery of messages to avoid delayed reactions in case of an accident. While the use of TSN for in-vehicle networks has already been investigated and is more mature, real-time communication with guaranteed latencies between vehicles and the cloud is still an open challenge.¶

Another important use case for the automotive industry is to further optimize the manufacturing process by taking more data sources into account. Since most SCADA systems and PLCs cannot connect to large data lakes or perform more advanced computational jobs, real-time communication between a cloud instance and the shop floor has several benefits. First, it integrates more data into the decision-making process, since control algorithms for automating manufacturing sites that run in a cloud environment can take more variables into account than single-plant control systems. Second, it reduces the resources needed on a particular factory floor or plant by transferring complex, recurring and resource-intensive computational jobs to a cloud provider, where computational power is not the limiting factor. However, in such cases it is important that the associated control can still occur in real time and according to very precise timing constraints. Such a development is already foreseen by trends such as Industry 4.0 and Control-as-a-Service (CaaS).¶

1.4. Background

The following background introduces and explains gLBF step by step. It uses IEEE 802.1 "Time Sensitive Networking" (TSN) mechanisms for reference, because at the time of writing, TSN bounded latency mechanisms are the most commonly understood and deployed mechanisms providing bounded latency and jitter services.¶

All mechanisms compared here, as well as those used by TSN, are based on an overall service design in which traffic flows have a well-defined rate and burstiness, which are tracked by the controller-plane and are called the "envelope" of the flow in this memo.¶

The traffic model used for gLBF is taken from UBS. It gives the flow a rate r [bits/sec] and a burst size b [bits], and the traffic envelope condition is that the total number of bits w(t) of the flow over a period t [sec] must be equal to or smaller than (r * t + b). In one typical case, a flow sends one packet of size b every interval p [sec]. This translates into a rate r = b / p for the flow because, after the flow has sent the first packet of b bits, it takes p seconds until (r * t) reaches b again: (r * t) = ((b / p) * p).¶

The controller-plane uses this per-flow information to calculate for each flow a path with sufficient free bandwidth and per-hop buffers so that the bounded end-to-end latency of packets can be guaranteed. It then allocates bandwidth and buffer resources to the flow so that further flows will not impact it.¶

Within this framework, bounded latency mechanisms can in general be divided into "on-time" and "in-time" mechanisms.¶

2.1. Dampers

The principle of the mechanism presented here is the so-called "Damper" mechanism, first described in the early 1990s but never standardized, primarily because the required forwarding was considered too advanced for the equipment of the time. These limitations are now starting to be resolved, so it seems like a good time to reintroduce the mechanism.¶

The principle of damper-based forwarding is easily explained: when node A sends a packet, it has measured the latency L, i.e., how long the packet was processed in the node. The main component of L is the queuing latency of the packet in A caused by competing traffic to the same outgoing interface. Based on the admission control and queuing algorithms used in the node, there can be a known upper bound M(ax) for this processing latency. When the packet then arrives at the next-hop node B, B simply delays the packet further by (M-L). As a result, the packet has been forwarded synchronously from A to B with a constant latency of M(ax).¶

+------------------------+      +------------------------+
| Node A                 |      | Node B                 |
|   +-+         +-+      |      |   +-+         +-+      |
|-x-|F|---------|Q|------|------|-x-|F|---------|Q|------|
|   +-+         +-+      | Link |   +-+         +-+      |
+------------------------+      +------------------------+
  |<--------- Hop A/B latency --->|

Figure 1 shows a single hop from node A to node B without Dampers.¶

A receives a packet. The F(orwarding) module determines the outgoing interface towards node B, where the packet is enqueued into Q, because competing packets may already be waiting to be sent from Q to B. This happens because packets may arrive simultaneously from multiple input interfaces on A (not shown in the figure), and/or because the speed of the receiving interface on A is higher than the speed of the link to B.¶

Forwarding F can be L2 switching and/or L3 routing; the choice does not impact the considerations described here.¶

When the packet is finally sent from Q over the link to B, B repeats the same steps towards the next node (not shown).¶

x is the reception time of the packet in A and B. The per-hop latency of the packet is xB - xA. It is predominantly determined by the time the packet has to wait in Q, plus any other relevant fixed or variable processing latency from F, plus the latency of the packet across the link. The latter is predominantly determined by the serialization latency of the packet plus the propagation latency of the link, which depends on the speed of light in the material used; in fiber, for example, light travels 31 percent slower than in vacuum.¶

All factors impacting the hop A/B latency other than Q in A are naturally bounded: a well-defined maximum can be calculated or conservatively estimated. The transmission latency of the packet from A to B is composed of the serialization latency of the packet, which can be calculated from the packet length and the bit rate of the interface, and the propagation latency through the link, which can be calculated from the speed of light in the link material and the length of the link.¶

To guarantee bounded latency through Q, an admission control system is required that tracks, accounts for and limits the total amount of traffic through Q. Admission control mechanisms rely on knowing the maximum burst that each traffic flow can cause, and add up those bursts to determine the maximum amount of simultaneous packet data in Q, which in turn determines the maximum latency of each individual packet through Q.¶

With such an admission control system, one can therefore calculate a maximum hop A/B latency MAX for a packet. However, packets will naturally have a variable hop A/B latency lower than MAX, due to differences in packet size and in competing traffic. As a result, their relative arrival times xB in B will differ from their relative arrival times xA in A. This leads to so-called "burst accumulation", and hence to the problem that the admission control system cannot calculate the maximum burst and latency through Q in B as easily as it can for Q in A.¶

If, however, packets could be forwarded such that their hop A/B latencies were all the same (synchronous, on-time), then the admission control system could apply the same calculus to B as to A. This is what damper mechanisms aim to achieve.¶

+------------------------+      +------------------------+
| Node A                 |      | Node B                 |
|   +-+   +-+   +-+      |      |   +-+   +-+   +-+      |
|-x-|D|-y-|F|---|Q|------|------|-x-|D|-y-|F|---|Q|------|
|   +-+   +-+   +-+      | Link |   +-+   +-+   +-+      |
+------------------------+      +------------------------+
        |<- A/B in-time latency ->|
        |<--A/B on-time latency ------->|

Figure 2 shows the Damper mechanism that is simplest to explain, but not to implement. Node A measures the time xA and sends this value in a packet header to B. B measures the time xB directly on reception of the packet. It then delays the packet for a time (MAX-(xB-yA)). As a result, the latency (yB-yA) will be exactly MAX, up to the accuracy of the damper.¶

The first challenge with this simple approach is the need to synchronize the clocks of A and B, so that (xB-yA) can be calculated correctly.¶

+------------------------+      +------------------------+
| Node A                 |      | Node B                 |
|   +-+   +-+   +-+      |      |   +-+   +-+   +-+      |
|-x-|D|-y-|F|---|Q|----z-|------|-x-|D|-y-|F|---|Q|----z-|
|   +-+   +-+   +-+      | Link |   +-+   +-+   +-+      |
+------------------------+      +------------------------+
        |<- A/B in-time latency ->|
        |<--A/B on-time latency ------->|

Figure 3 shows how this can be resolved by also measuring the time at z. A calculates d = (MAX1-(zA-yA)) and sends d in a header of the packet to B. B then delays the packet in D by d relative to the locally measured time xB. Because neither the calculation of d in A nor the delay by d in D depends on clock synchronization between A and B, this approach eliminates the need for clock synchronization.¶

As a consequence of this change, however, the latency incurred by transmitting the packet over the link is no longer included in the measurement.¶

In this approach, MAX1 is the bounded latency for all processing of a packet except the link propagation latency P, which is the length of the medium divided by the speed of light in the medium, and MAX = MAX1 + P.¶

The serialization latencies across the sending interface on A and the receiving interface on B can, however, be included in MAX1. It is only necessary for the measured timestamps zA and xB to logically refer to the same point in time, assuming a link of minimal length, e.g., a back-to-back connection. For example, the reference time could be the time when the first bit of the packet can be observed on the shortest wire between A and B. A can most likely measure zA only some fixed time offset o before that reference time, and hence needs to correct zA += o before calculating d. Likewise, B may only be able to measure xB after the whole packet has arrived. This would be l * r later than the reference time, where r is the serialization rate of the receiving interface on B and l is the length of the packet. B would hence need to adjust d -= l * r to subtract this time from the time the packet needs to be delayed in D.¶

+------------------------+      +------------------------+
| Node A                 |      | Node B                 |
| +-+    +-+   +-+   +-+ |      | +-+    +-+   +-+   +-+ |
|-|F|--x-|D|-y-|Q|-z-|M|-|------|-|F|--x-|D|-y-|Q|-z-|M|-|
| +-+    +-+   +-+   +-+ |      | +-+    +-+   +-+   +-+ |
+------------------------+      +------------------------+
       |<- Dampened Q ->|              |<- Dampened Q ->|
             |<- A/B in-time latency ->|
             |<--A/B on-time latency ------->|

Figure 4 shows a further refined model for simpler implementation. Larger routers/switches are typically modular, and forwarding happens on a different modular component (such as a linecard) than the queuing for the outgoing interface. Requiring clock synchronization even within such a large device is undesirable.¶

Therefore, Figure 4 shows damper operations occurring solely before enqueuing packets into Q and after dequeuing them. The Damper module measures the timestamp x, extracts d from the packet header, adjusts it according to the considerations described above, delays the packet by that value and then enqueues it. After the packet is dequeued, the Marking module measures the time z, adjusts it according to the same considerations, calculates the value of d and rewrites it in the packet header before sending the packet.¶

2.2. Dampers and controller-plane

         Controller-plane:
        Path Control  and
         Admission Control
      ..     .      .      ..
     .       .       .       .
   +---+   +---+   +---+   +---+  Src---| A |---| B |---| C |---| D |
   +---+   +---+   +---+   +---+
     |       |       |       |
   +---+   +---+   +---+   +---+
   | E |---| F |---| G |---| H |--- Dst
   +---+   +---+   +---+   +---+

While the damper mechanism described so far can be realized with different queuing mechanisms, and hence different calculi, on every interface that supports bounded latency, the role of the controller-plane involves other components besides admission control, and those need to be tightly coupled with admission control for efficient use of resources.¶

Figure 5 illustrates the most common problem. For a new traffic flow from Src to Dst, the controller-plane needs to provide a path through the network and reserve the resources along it. The simplest form, for bandwidth reservation only, is called Constrained Shortest Path First (CSPF). When end-to-end latency guarantees also need to be provided in support of bounded latency, not only does the path calculation become more complex, but many per-hop bounded latency queuing mechanisms also support more than one per-hop latency on each hop, and the controller-plane needs to determine which one is best for each hop of a candidate path.¶

As a result, to perform these operations, the controller-plane needs to know exactly which parameters the bounded latency queuing mechanism on each hop/interface supports. Standardizing these parameters ultimately means standardizing the bounded latency queuing mechanism, and not only the damper part.¶

3.1. Damper with UBS queuing/calculus

Guaranteed Latency Based Forwarding (gLBF) uses the queuing model of Urgency Based Scheduling (UBS), which is also used in TSN Asynchronous Traffic Shaping (TSN-ATS). This allows one and the same controller-plane, including its optimization algorithms for bandwidth and latency control, to be reused or co-developed for both TSN-ATS (or a DetNet L3 equivalent) and gLBF. Hence, gLBF should provide the most easily operationalized on-time solution for networks that want to evolve from an in-time model based on TSN-ATS, or that even want to operate both options in parallel.¶

Effectively, gLBF replaces the per-flow interleaved regulators of UBS with per-flow stateless damper operations. Whereas UBS only provides in-time latency guarantees, gLBF as a result provides on-time latency service, but with fundamentally the same calculus as UBS.¶

   +-----------------------------------------------+
   | UBS strict priority Queuing block             |
   |                                               |
   |               +--------------+   +----------+ |
   | +-------+   /-| Prio 1 queue |---|          | |
   | | Packet|  /  +--------------+   | Strict   | |
-->| | Prio  |->         ...         -| Priority | |--->
   | | enque |  \  +--------------+   | Schedule | |
   | +-------+   \-| Prio 8 queue |---|          | |
   |               +--------------+   +----------+ |
   +-----------------------------------------------+

Strict priority scheduling always takes the next packet from the highest-priority queue (1 is highest) that has a pending packet, and forwards it. UBS defines two options for the traffic model of traffic flows. In the more flexible one, each flow specifies a rate r and a maximum burst size b (in bits).¶

As a result, the bounded latency of a packet in priority 1 is (roughly) the latency required to serialize the sum of the bursts of all the flows admitted into priority 1. The bounded latency of a packet in priority 2 is the priority 1 latency plus the latency required to serialize the sum of the bursts of all the flows admitted into priority 2, and so on.¶

3.2. gLBF processing

The following text extends/refines the damper processing as necessary for gLBF.¶

   +------------------------+       +------------------------+
   | Node A                 |       | Node B                 |
   | +-+    +-+   +-+   +-+ |       | +-+    +-+   +-+   +-+ |
 --|-|F|--x-|D|-y-|Q|-z-|M|-|-------|-|F|--x-|D|-y-|Q|-z-|M|-|--->
IIF| +-+    +-+   +-+   +-+ |OIF IIF| +-+    +-+   +-+   +-+ |OIF
   +------------------------+       +------------------------+
          |<- Dampened Q ->|               |<- Dampened Q ->|
                |<- A/B in-time latency -->|
                |<--A/B on-time latency -------->|

The Damper module (D) retrieves the delay from a packet header field. Requirements for that header field are defined in Section 3.4.¶

Because serialization speeds differ between Incoming InterFaces (IIF), and because D will likely need to compensate the measured time x based on the packet length and serialization speed, an internal packet header should carry this information. Note that this is solely an implementation consideration and should not impact the configuration model of gLBF.¶

The Queuing module is logically as described in UBS, except that the priority of a packet in gLBF is not selected based on per-flow state, but from an appropriate packet header field, which is looked up in the Packet Prio Enque stage of Q; the packet is then enqueued according to that field's value. Possible options for indicating such a priority in a packet header field are defined below in Section 3.4.¶

Like Q, M needs to look up the packet priority to determine MAX1 for the packet, and hence calculate the value d that it rewrites in the packet header.¶

The overall configuration data model to be defined for gLBF is hence the configuration data model of UBS, which is a set of priority queues and their maximum sizes, plus, for gLBF, a maximum latency value MAX1 for each of these queues, calculated by the controller-plane and used by the M module.¶

Given that the node knows the serialization speed of the OIF, MAX1 for each queue could be derived automatically from the maximum length in bytes of each UBS priority queue. Additional configuration parameters to fine-tune the latency, such as increasing it by some percentage as a safety margin, might be beneficial though. Such implementation details are outside the scope of this specification.¶

TBD: sketch a small YANG-like data model, even though it is trivial, as in the TCQF draft. It should primarily include additional diagnostic elements, such as the maximum occupancy of each priority queue and the number of overruns.¶

3.3. Error handling

A well-specified standard for a damper mechanism such as the one described in this document also needs to handle error cases. The primary error condition is when the sending node A of a gLBF packet recognizes that the packet was enqueued for longer than MAX1, so that the delay to be written into the packet would have to be < 0.¶

In this case, error signaling such as ICMP/ICMPv6 needs to be triggered (rate-limited!), and the packet needs to be discarded, to avoid admission control failures or congestion for other packets further down the path.¶

The data model below describes an option that, instead of discarding the packet, signals the condition through an ECN-like mechanism combined with lower-priority forwarding of the packet. This informs the receiver directly about the problem and allows the application to deal with such conditions better than through other error signaling.¶

3.4. Data Model for gLBF packet metadata

gLBF is specifically designed to support per-hop, per-flow stateless operation, because its processing requires no per-flow metadata, only per-packet metadata.¶

3.5. Ingress and Egress processing

4.1. gLBF versus UBS / TSN-ATS

By relying on [UBS] for both the traffic model and the bounded latency calculus, gLBF should be easy to operationalize using existing controller-plane implementations for TSN-ATS: UBS/TSN-ATS and gLBF provide the same bounded latency and use the same model to manage bandwidth for different flows, namely calculating which flow goes into which priority queue on which hop.¶

The main change to a UBS/TSN-ATS controller-plane when using gLBF concerns what happens when a flow is admitted into the network, removed, or rerouted. In UBS, each of these operations requires the controller-plane to signal the traffic flow parameters to each node along the path, so that the forwarding plane can establish the per-hop, per-flow state for the flow. Depending on the network configuration, this also implies configuring the next hop of the flow for routing.¶

For gLBF hop-by-hop operations, the first hop needs to receive the per-path or per-hop priority information, which it then imposes in an appropriate packet header of the flow's packets.¶

4.2. First-hop policing

In gLBF, the controller-plane has to perform this action only towards the ingress node of the gLBF domain. Traffic parameters such as rate and burst size are only needed there to establish a policer, so that the flow cannot violate its admitted traffic parameters. This first-hop policing is actually a function independent of gLBF, UBS or any other method used along the path.¶

4.3. Path steering

gLBF operates independently of the path steering mechanism, but the controller-plane will very likely want to ensure that gLBF (or, for that matter, any DetNet) traffic does not unexpectedly get rerouted by in-network routing mechanisms, because it normally will not, or cannot, reserve resources for such rerouted flows on arbitrary failure paths without significant additional effort and/or waste of resources.¶

4.4. Decentralized controller-plane

Today's (pre-DetNet) controller-plane deployments commonly use one centralized "SDN" controller to implement the functions of the "Path Computation Element" (PCE), which tracks free bandwidth on every hop in the network topology and can hence calculate on which path enough resources are available for a requested new flow. This is, for example, the typical deployment with RSVP-TE and Segment Routing.¶

However, it is equally possible to design a fully decentralized controller-plane, especially for DetNet networks that may not have all the complex policy requirements that make a centralized PCE most desirable. This is possible because, with gLBF, the controller-plane does not need to interact with non-ingress routers when dynamically allocating resources to new traffic flows.¶

The model for a fully decentralized controller-plane could be derived from the model developed and deployed for RSVP-TE without a centralized PCE, where the ingress routers (LSRs) perform path calculations based on IGP-flooded information about the available resources on every hop. However, the necessary protocol operations may not lend themselves well to existing IGPs, and a new decentralized controller-plane may even need to be developed.¶

Note, though, that none of these (complex) controller-plane considerations need to apply to deployments where the total amount of gLBF traffic is well defined and so small compared to the available capacity that no bandwidth exhaustion would occur, even under supported failure conditions. In those cases, gLBF is still necessary to guarantee the latency of the traffic, but the whole controller-plane is reduced to a network capacity planning and ingress-router configuration task. This "controller-plane-free" mode of operation is standard practice in many large service provider networks for traffic that requires guaranteed bandwidth but has statically known (limited) capacity requirements, such as IPTV.¶

8.1. Normative References

[RFC2210]

Wroclawski, J., "The Use of RSVP with IETF Integrated Services", RFC 2210, DOI 10.17487/RFC2210, September 1997, <https://www.rfc-editor.org/rfc/rfc2210>.

[RFC2212]

Shenker, S., Partridge, C., and R. Guerin, "Specification of Guaranteed Quality of Service", RFC 2212, DOI 10.17487/RFC2212, September 1997, <https://www.rfc-editor.org/rfc/rfc2212>.

[RFC2474]

Nichols, K., Blake, S., Baker, F., and D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, DOI 10.17487/RFC2474, December 1998, <https://www.rfc-editor.org/rfc/rfc2474>.

[RFC3270]

Le Faucheur, F., Ed., Wu, L., Davie, B., Davari, S., Vaananen, P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi-Protocol Label Switching (MPLS) Support of Differentiated Services", RFC 3270, DOI 10.17487/RFC3270, May 2002, <https://www.rfc-editor.org/rfc/rfc3270>.

[RFC8200]

Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", STD 86, RFC 8200, DOI 10.17487/RFC8200, July 2017, <https://www.rfc-editor.org/rfc/rfc8200>.

[RFC8655]

Finn, N., Thubert, P., Varga, B., and J. Farkas, "Deterministic Networking Architecture", RFC 8655, DOI 10.17487/RFC8655, October 2019, <https://www.rfc-editor.org/rfc/rfc8655>.

[RFC8964]

Varga, B., Ed., Farkas, J., Berger, L., Malis, A., Bryant, S., and J. Korhonen, "Deterministic Networking (DetNet) Data Plane: MPLS", RFC 8964, DOI 10.17487/RFC8964, January 2021, <https://www.rfc-editor.org/rfc/rfc8964>.

8.2. Informative References

[BIER-TE]

Eckert, T. T., Menth, M., and G. Cauchie, "Tree Engineering for Bit Index Explicit Replication (BIER-TE)", Work in Progress, Internet-Draft, draft-ietf-bier-te-arch-13, 25 April 2022, <https://datatracker.ietf.org/doc/html/draft-ietf-bier-te-arch-13>.

[CQF]

IEEE Time-Sensitive Networking (TSN) Task Group., "IEEE Std 802.1Qch-2017: IEEE Standard for Local and Metropolitan Area Networks - Bridges and Bridged Networks - Amendment 29: Cyclic Queuing and Forwarding (CQF)", 2017.

[dgLBF]

Massa, J., Forti, S., Paganelli, F., Dazzi, P., Brogi, A., Clemm, A., and T. Eckert, "Declarative traffic engineering for Low-Latency and reliable networking", Future Generation Computer Systems, Vol. 182, 108494, ISSN 0167-739X, DOI 10.1016/j.future.2026.108494, 2026, <https://doi.org/10.1016/j.future.2026.108494>.

[DNBL]

Finn, N., Le Boudec, J., Mohammadpour, E., Zhang, J., and B. Varga, "Deterministic Networking (DetNet) Bounded Latency", Work in Progress, Internet-Draft, draft-ietf-detnet-bounded-latency-10, 8 April 2022, <https://datatracker.ietf.org/doc/html/draft-ietf-detnet-bounded-latency-10>.

[gLBF]

Eckert, T., Clemm, A., and S. Bryant, "gLBF: Per-Flow Stateless Packet Forwarding with Guaranteed Latency and Near-Synchronous Jitter,", IEEE 2021 17th International Conference on Network and Service Management (CNSM), Izmir, Turkey, doi 10.23919/CNSM52442.2021.9615538, October 2021, <https://dl.ifip.org/db/conf/cnsm/cnsm2021/1570754857.pdf>.

[gLBF-Springer2023]

Eckert, T., Clemm, A., and S. Bryant, "High Precision Latency Forwarding for Wide Area Networks Through Intelligent In-Packet Header Processing (gLBF)", Springer Journal of Network and Systems Management, 31, Article number: 34 (2023), doi https://doi.org/10.1007/s10922-022-09718-9, February 2023.

[I-D.dang-queuing-with-multiple-cyclic-buffers]

Liu, B. and J. Dang, "A Queuing Mechanism with Multiple Cyclic Buffers", Work in Progress, Internet-Draft, draft-dang-queuing-with-multiple-cyclic-buffers-00, 22 February 2021, <https://datatracker.ietf.org/doc/html/draft-dang-queuing-with-multiple-cyclic-buffers-00>.

[I-D.eckert-detnet-bounded-latency-problems]

Eckert, T. T. and S. Bryant, "Problems with existing DetNet bounded latency queuing mechanisms", Work in Progress, Internet-Draft, draft-eckert-detnet-bounded-latency-problems-00, 12 July 2021, <https://datatracker.ietf.org/doc/html/draft-eckert-detnet-bounded-latency-problems-00>.

[I-D.eckert-detnet-flow-interleaving]

Eckert, T. T., "Deterministic Networking (DetNet) Data Plane - Flow interleaving for scaling detnet data planes with minimal end-to-end latency and large number of flows.", Work in Progress, Internet-Draft, draft-eckert-detnet-flow-interleaving-03, 7 July 2025, <https://datatracker.ietf.org/doc/html/draft-eckert-detnet-flow-interleaving-03>.

[I-D.ietf-detnet-dataplane-taxonomy]

Joung, J., Geng, X., Peng, S., and T. T. Eckert, "Dataplane Enhancement Taxonomy", Work in Progress, Internet-Draft, draft-ietf-detnet-dataplane-taxonomy-07, 1 November 2026, <https://datatracker.ietf.org/doc/html/draft-ietf-detnet-dataplane-taxonomy-07>.

[I-D.stein-srtsn]

Stein, Y. J., "Segment Routed Time Sensitive Networking", Work in Progress, Internet-Draft, draft-stein-srtsn-01, 29 August 2021, <https://datatracker.ietf.org/doc/html/draft-stein-srtsn-01>.

[IEEE802.1Q]

IEEE 802.1 Working Group, "IEEE Standard for Local and Metropolitan Area Network — Bridges and Bridged Networks (IEEE Std 802.1Q)", doi 10.1109/ieeestd.2018.8403927, 2018, <https://doi.org/10.1109/ieeestd.2018.8403927>.

[IEEE802.1Qbv]

IEEE Time-Sensitive Networking (TSN) Task Group., "IEEE Standard for Local and metropolitan area networks -- Bridges and Bridged Networks - Amendment 25: Enhancements for Scheduled Traffic (TAS)", 2015.

[IPV6-PARMS]

"Internet Protocol Version 6 (IPv6) Parameters", IANA , n.d., <https://www.iana.org/assignments/ipv6-parameters/ipv6-parameters.xhtml>.

[LBF]

Eckert, T. and A. Clemm, "High-Precision Latency Forwarding over Packet-Programmable Networks", IEEE 2020 IEEE/IFIP Network Operations and Management Symposium (NOMS 2020), doi 10.1109/NOMS47738.2020.9110431, April 2020.

[LDN]

Liu, B., Ren, S., Wang, C., Angilella, V., Medagliani, P., Martin, S., and J. Leguay, "Towards Large-Scale Deterministic IP Networks", IEEE 2021 IFIP Networking Conference (IFIP Networking), doi 10.23919/IFIPNetworking52078.2021.9472798, 2021, <https://dl.ifip.org/db/conf/networking/networking2021/1570696888.pdf>.

[LSDN]

Qiang, L., Geng, X., Liu, B., Eckert, T. T., Geng, L., and G. Li, "Large-Scale Deterministic IP Network", Work in Progress, Internet-Draft, draft-qiang-detnet-large-scale-detnet-05, 2 September 2019, <https://datatracker.ietf.org/doc/html/draft-qiang-detnet-large-scale-detnet-05>.

[multipleCQF]

Finn, N., "Multiple Cyclic Queuing and Forwarding", October 2021, <https://www.ieee802.org/1/files/public/docs2021/new-finn-multiple-CQF-0921-v02.pdf>.

[RFC3209]

Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, <https://www.rfc-editor.org/rfc/rfc3209>.

[RFC4875]

Aggarwal, R., Ed., Papadimitriou, D., Ed., and S. Yasukawa, Ed., "Extensions to Resource Reservation Protocol - Traffic Engineering (RSVP-TE) for Point-to-Multipoint TE Label Switched Paths (LSPs)", RFC 4875, DOI 10.17487/RFC4875, May 2007, <https://www.rfc-editor.org/rfc/rfc4875>.

[RFC8296]

Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A., Tantsura, J., Aldrin, S., and I. Meilik, "Encapsulation for Bit Index Explicit Replication (BIER) in MPLS and Non-MPLS Networks", RFC 8296, DOI 10.17487/RFC8296, January 2018, <https://www.rfc-editor.org/rfc/rfc8296>.

[RFC8402]

Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., Decraene, B., Litkowski, S., and R. Shakir, "Segment Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, July 2018, <https://www.rfc-editor.org/rfc/rfc8402>.

[RFC8754]

Filsfils, C., Ed., Dukes, D., Ed., Previdi, S., Leddy, J., Matsushima, S., and D. Voyer, "IPv6 Segment Routing Header (SRH)", RFC 8754, DOI 10.17487/RFC8754, March 2020, <https://www.rfc-editor.org/rfc/rfc8754>.

[RFC8938]

Varga, B., Ed., Farkas, J., Berger, L., Malis, A., and S. Bryant, "Deterministic Networking (DetNet) Data Plane Framework", RFC 8938, DOI 10.17487/RFC8938, November 2020, <https://www.rfc-editor.org/rfc/rfc8938>.

[RFC8986]

Filsfils, C., Ed., Camarillo, P., Ed., Leddy, J., Voyer, D., Matsushima, S., and Z. Li, "Segment Routing over IPv6 (SRv6) Network Programming", RFC 8986, DOI 10.17487/RFC8986, February 2021, <https://www.rfc-editor.org/rfc/rfc8986>.

[RFC9016]

Varga, B., Farkas, J., Cummings, R., Jiang, Y., and D. Fedyk, "Flow and Service Information Model for Deterministic Networking (DetNet)", RFC 9016, DOI 10.17487/RFC9016, March 2021, <https://www.rfc-editor.org/rfc/rfc9016>.

[RFC9262]

Eckert, T., Ed., Menth, M., and G. Cauchie, "Tree Engineering for Bit Index Explicit Replication (BIER-TE)", RFC 9262, DOI 10.17487/RFC9262, October 2022, <https://www.rfc-editor.org/rfc/rfc9262>.

[RFC9320]

Finn, N., Le Boudec, J.-Y., Mohammadpour, E., Zhang, J., and B. Varga, "Deterministic Networking (DetNet) Bounded Latency", RFC 9320, DOI 10.17487/RFC9320, November 2022, <https://www.rfc-editor.org/rfc/rfc9320>.

[SCQF]

Chen, M., Geng, X., Li, Z., Joung, J., and J. Ryoo, "Segment Routing (SR) Based Bounded Latency", Work in Progress, Internet-Draft, draft-chen-detnet-sr-based-bounded-latency-03, 7 July 2023, <https://datatracker.ietf.org/doc/html/draft-chen-detnet-sr-based-bounded-latency-03>.

[TCQF]

Eckert, T. T., Li, Y., Bryant, S., Malis, A. G., Ryoo, J., Liu, P., Li, G., and S. Ren, "Deterministic Networking (DetNet) Data Plane - Tagged Cyclic Queuing and Forwarding (TCQF) for bounded latency with low jitter in large scale DetNets", Work in Progress, Internet-Draft, draft-eckert-detnet-tcqf-12, 15 December 2025, <https://datatracker.ietf.org/doc/html/draft-eckert-detnet-tcqf-12>.

[TSN-ATS]

Specht, J., "P802.1Qcr - Bridges and Bridged Networks Amendment: Asynchronous Traffic Shaping", IEEE , 9 July 2020, <https://1.ieee802.org/tsn/802-1qcr/>.

[UBS]

Specht, J. and S. Samii, "Urgency-Based Scheduler for Time-Sensitive Switched Ethernet Networks", IEEE 28th Euromicro Conference on Real-Time Systems (ECRTS), 2016.

A.1. High speed example pseudocode instead of reference pseudocode

This memo does not provide normative reference code aligned with the normative functional description in Section 3. The primary reason is that a naive 1:1 implementation of the functional description would lead to inferior performance. Nevertheless, the feasibility of high-speed implementations is an important factor in the deployability of mechanisms like gLBF. Therefore, this appendix describes considerations for such high-speed implementations, including pseudocode for one possible option.¶

[gLBF] describes two possible optimized hardware implementation approaches for gLBF: one using "Push In First Out" (PIFO) queues, the other using timed FIFO queues. The following describes and explains the second approach, because it is hopefully more feasible for nearer-term hardware implementations, given that scalable high-speed PIFO hardware implementations are still a matter of research.¶

However, the timed FIFO algorithm presented here may also face hardware scalability challenges, because it requires O(IIF * priorities^2) FIFO queues per outgoing interface, where IIF is the number of (incoming) interfaces on the node and priorities is the number of desired priority levels (TSN allows up to 8). If, however, the priority of flows (packets) is not per-hop but the same on every hop (per-path), then the number is just (IIF * priorities).¶

A.2. UBS high speed implementations

The algorithm for the pseudocode shown later in this appendix is based on the analysis of gLBF behavior performed in [gLBF]. That analysis reapplies the same type of analysis and algorithm as [UBS], which can be used to implement TSN-ATS in high-speed switching hardware. Therefore, this appendix starts by explaining the UBS mechanisms.¶

UBS (see [UBS], Figure 4) logically consists of two separate stages. The first stage is a per-priority, per-incoming-interface (IIF) interleaved regulator. An interleaved regulator is a FIFO where dequeuing of the queue head packet (qhead) is based on the per-flow state of qhead.¶

A target departure time for the qhead is calculated from the flow state that the router maintains across packets, and once that departure time is reached, the packet is passed to one of the egress strict priority queues. It is called an interleaved regulator because the FIFO holds packets from multiple flows (all from the same incoming interface and with the same outgoing priority), hence 'interleaved', and 'regulator' because it delays the packet until the target departure time.¶

The innovation of interleaved regulators, compared to the earlier per-flow shapers as used in [RFC2210], is the recognition and mathematical proof that the arrival and target departure times of all packets received from the same IIF (prior-hop node) with the same egress priority are in the same order, so that these packets can be enqueued into a single FIFO in which per-flow processing only needs to happen for the qhead, with no more than negligible added latency compared to per-flow shapers.¶

With this understanding, the interleaved regulator and the following strict priority stage can easily be combined into a single queuing stage with appropriate scheduling. In this "flattened" approach, each logical egress strict priority queue consists of the per-IIF interleaved regulators (FIFO queues) for that priority.¶

Egress scheduling now needs to perform the per-flow state processing for the qhead of each queue, and then take the target departure time of that packet into account, selecting the interleaved regulator whose qhead has the highest priority and earliest target departure time.¶

A.3. gLBF compared to UBS

The same line of thought as in UBS can be applied to gLBF and is the basis for the algorithm and pseudocode described here.¶

In gLBF, the order of packets by arrival and target departure time depends on the egress priority and the IIF, but also on the priority the packet had on the prior hop (pprio). All packets with the same (IIF,prio,pprio) arrive in the same order in which they need to depart, so it is possible to enqueue them into a FIFO and only perform timed dequeuing of the qhead.¶

Both prio and pprio matter because, when gLBF uses an encapsulation that allows per-hop priority indications, a packet's priority can differ on each hop. In UBS, the priority of a packet can equally differ across hops, but the prior-hop priority is not needed to separate packets into interleaved regulators, because UBS targets in-time delay, where the interleaved regulator only needs to compensate for burst accumulation. In gLBF, in contrast, the delay to be introduced depends on the damper value, which depends on the prior-hop priority. As a result, the order in which gLBF packets should depart (absent any burst accumulation) also depends on the prior-hop priority.¶

Because gLBF does not deal with per-flow state on the router as UBS does in its interleaved regulators, the FIFOs for gLBF are called timed FIFOs in this memo rather than interleaved regulators.¶

A.4. Comparison to normative behavior

Whereas the normative description defines D, Q and M blocks, where the D block performs the dampening and the Q block performs the priority queuing, the high-speed hardware optimization replaces D and Q with a single DQ block. In it, packets are enqueued into a single-stage per-(IIF,pprio,prio) FIFO (similar to UBS), and the dampening by the damper time happens on dequeuing from that stage (also similar to UBS), while packets are scheduled/prioritized based on their prio.¶

   +------------------------+       +------------------------+
   | Node A                 |       | Node B                 |
   | +-+    +-------+   +-+ |       | +-+    +-------+   +-+ |
 --|-|F|--x-| DQ y  |-z-|M|-|-------|-|F|--x-| DQ y  |-z-|M|-|--->
IIF| +-+    +-------+   +-+ |OIF IIF| +-+    +-------+   +-+ |OIF
   +------------------------+       +------------------------+
          |<- Dampened Q ->|               |<- Dampened Q ->|
                 |<- A/B in-time latency ->|
                 |<--A/B on-time latency -------->|

In the normative definition, the damper value to be put into the packet depends on the time y at which the packet leaves the damper stage D and enters the priority queue stage Q. Simplified, the damper value is (MAX1 - (z - y)).¶

In the high-speed implementation, packets are not passed from a damper stage queue to a priority queue; instead, they stay in the same queue. On the one hand, this is one of the core reasons why this approach can support high speed: it does not require two-stage enqueuing/dequeuing. On the other hand, it also eliminates the ability to take a timestamp at time y.¶

To replicate the normative gLBF behavior with a single stage, the marking stage M (where the new damper value is calculated) therefore needs to know y by other means.¶

Fortunately, y is exactly the target departure time of the packet from the D stage, and therefore also from the DQ stage, so it already needs to be calculated on entry to DQ: y = (x + damper - <details>), as explained below.¶

The pseudocode consists of glbf_enqueue() describing the enqueuing into the DQ stage, and glbf_dequeue() describing the dequeuing from the DQ stage. glbf_dequeue() (synchronously) calls glbf_send() which represents the marking stage M.¶

A.5. Pseudocode

void glbf_enqueue(pak,oif) {
  tdamp = pak.header.tdamp // damper value from packet
  prio  = pak.header.prio  // this hops packet prio
  pprio = pak.header.pprio // previous hops packet prior
  iif   = pak.context.iif  // incoming interface of packet

  ta = adj_rcv_time(now(),                  \[1]
                    pak.context.iif.speed,
                    pak.context.length)

  td = now() + ta // (dampened earliest) departure time
  pak.context.max1 = max1\[oif,prio]
  enqueue(pak, td, q(oif,iif,prio,pprio))
}

time_t adj_rcv_time(now, speed, length) {
  time_t now
  int speed, length

  return now - length * 8 / speed - FUDGE
}

void glbf_dequeue(oif) {
  next_packet: while(1) {
    tnow = now()
    ftd = tnow + 1  // time of first packet to send
    fq = NULL      // queue of first packet to send
    foreach prio in maxpriority...minpriority {
      foreach iif in iifs {
        foreach pprio in priorities}
          // find qhead with earliest departure time
          if (q = q(oif,iif,prio,pprio).qhead) {
            td = q.qhead.td   // target (departure) time
            if td < ftd
              ftd = td
              fq = q
            }
          }
        }
      }
      if fq { // found packet
        pak = dequeue(q,oif)
        glbf_send(oif,tnow,pak)
        break next_packet
      }
    }
  }
}

void glbf_send(oif,tnow,pak) {
  qtime = tnow - pak.context.td          // \[1]
  td = pak.context.max1 - qtime - FUDGE2 // \[2]
  pak.header.tdamp = td
  serialize(oif,pak)
}

A.6. Performance considerations

PTP clock synchronization needs to capture the accurate arrival time of PTP packets. It also measures the residence time of PTP packets between receiving and sending them with nsec accuracy, and adds it to a PTP packet header field (correctionField). Given that this needs to be hardware-supported functionality, it is unclear whether there is a relevant difference between supporting it for a few PTP signaling packets and for a much larger number of gLBF data packets, or whether it is a good indication of the feasibility of the packet processing steps that gLBF requires.¶

C.1. Solution scope

gLBF is not proposed as a standalone solution, but in conjunction with [I-D.eckert-detnet-flow-interleaving], a per-flow mechanism used on the ingress gLBF router to allow more flexible per-flow management of the admission of flow bursts than presented in this document. The proposal is split into two documents because [I-D.eckert-detnet-flow-interleaving] can be combined with all "mostly synchronous" per-hop mechanisms, for example, besides gLBF, also CQF and [TCQF].¶

C.2. Taxonomy

Using the section numbers/titles of the taxonomy document [I-D.ietf-detnet-dataplane-taxonomy], gLBF can be classified as follows. In summary, gLBF is very much like TCQF, except that it replaces cycles with Dampers, thereby further reducing end-to-end jitter, reducing clock accuracy requirements (only phase synchronization, not clock synchronization, is required) and removing the complexity of end-to-end cycle management. Compared to older Damper-based proposals, gLBF reuses the ATS calculus, which allows reuse of ATS admission control and delay calculations, and a simple implementation based on FIFOs with timed scheduling (like ATS).¶

• 4.1 Per Hop Dominant Factor for Latency Bound¶

  The per-hop dominant latency bound is Category 3. The per-hop latency for each class is determined by the configured burst size.¶

• 5.1 Periodicity¶

  gLBF is not periodic. Its fundamental mode of operation is not based on cycles to manage bursts, but on per-hop, per-packet synchronous delivery, which maintains burst sizes end-to-end across all hops.¶

• 5.2. Traffic Granularity¶

  Formally, gLBF has class-level granularity. However, the statement in [I-D.ietf-detnet-dataplane-taxonomy] (revision 04) that all packets within a class are treated the same is questionable for gLBF (and for Damper solutions in general): in Damper solutions like gLBF, packet latency is calculated per packet (within the same class) based on the previously experienced per-hop latency, in order to compensate for it.¶

• 5.3. Time Bounds¶

  gLBF is per-hop bounded. It is Right Bounded by the configured burst size for the hop and class of the packet. It is Left Bounded by the signaled remaining latency needed to achieve synchronous forwarding. If the bound were observed not on the wire, but at a specific forwarding step in the router (after reception, before enqueuing), then the per-hop bound would be extremely tight, limited only by implementation accuracy.¶

• 5.4 Service Order¶

  gLBF service order is solely arrival based.¶

• 1. Suitable Categories for DetNet¶

  gLBF is "6.5. Class level non-periodic bounded category" according to the categorization.¶

C.3. Status

gLBF primarily has an open-source, simulation-level validation implementation, as described in [gLBF]. Additional implementations were done in the Linux kernel by a university (TBD reference) and by a second university in conjunction with a Prolog-based admission control server implementation ([dgLBF]).¶