State Synchronization Procedures between Stateful Path Computation Elements (PCEs)

Introduction The Path Computation Element communication Protocol (PCEP) provides mechanisms for Path Computation Elements (PCEs) to perform path computations in response to Path Computation Clients' (PCCs) requests. A stateful PCE is capable of considering, for the purposes of path computation, not only the network state in terms of links and nodes (referred to as the Traffic Engineering Database or TED), but also the status of active services (previously computed paths), and currently reserved resources, stored in the Label Switched Paths Database (LSP-DB). describes general considerations for a stateful PCE deployment and examines its applicability and benefits, as well as its challenges and limitations through a number of use cases. A PCC can synchronize LSP state information to a stateful PCE. The stateful PCE extensions allow a redundancy scenario where a PCC can have redundant PCEP sessions toward multiple PCEs. In such a case, a PCC gives control of an LSP to a single PCE, and only one PCE is responsible for path computation for this delegated LSP. does not specify any procedures for the exchange of LSP state between the PCEs themselves. This document describes procedures for synchronizing LSP state between peer stateful PCEs across various deployment scenarios, and for preventing computation loops. This is distinct from the communication between a child PCE and a parent PCE in the Hierarchical PCE architecture (see ). The procedures apply to LSPs of any path setup type . The problem is described in , the solution is outlined in , and the procedures and protocol extensions are specified in . This document is published as Experimental. The computation priorities that determine which PCE computes are provisioned by the operator rather than advertised in PCEP, and each PCE decides on sub-delegation from its own view of the other PCEs. With two PCEs, both have the same view. With three or more, a PCE cannot observe the sessions between the other PCEs, so their views may differ for a time. In that case the computation may be less optimal, but no loop can occur (). There are no known implementations. The purpose of the experiment is to find out whether this approach is sufficient for sets of more than two PCEs, or whether the priorities need to be advertised in PCEP together with an explicit role change procedure. Deployments SHOULD keep the set of synchronised PCEs as small as their redundancy or scaling objective allows ().

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Terminology This document uses the terminology defined in , , and . In addition, the following terms are used:

State-sync session:: A PCEP session between two stateful PCEs for which the state-sync capability () has been negotiated and to which the procedures in this document apply.
State-sync PCE set:: A set of stateful PCEs connected by a full mesh of state-sync sessions, sharing LSP state for the same set of PCCs, LSPs, or association groups.
Delegated PCE:: The PCE to which a PCC has delegated an LSP (D flag set in the PCRpt sent by the PCC) . The PCC interacts only with this PCE for that LSP.
Sub-delegation:: A delegated PCE passing control of an LSP to another PCE in the state-sync PCE set over a state-sync session, by setting the D flag in the forwarded PCRpt (). Sub-delegation is not visible to the PCC.
Computing PCE:: For a given LSP, PCC, or association group, the PCE with the highest computation priority () among the reachable members of the state-sync PCE set. It computes paths for those LSPs, whether delegated directly by the PCC or sub-delegated by another PCE.
Backup PCE:: Any PCE in the state-sync PCE set other than the computing PCE. A backup PCE collects and forwards state and can take over computation according to its computation priority.

Problem Statement There are some use cases where inter-PCE state synchronization can bring additional resiliency to the design, for instance, when some PCC-PCE session fails. The inter-PCE state synchronization may also provide a faster update of the LSP state when such an event occurs. When, in a redundant PCE scenario, there is a need to compute a set of paths that are part of a group (so there is a dependency between the paths), there may be some cases where the computation of all paths in the group is not handled by the same PCE: this situation is called a split-brain. This split-brain scenario may lead to computation loops between PCEs or suboptimal path computation. Finally, synchronizing state between PCEs allows the maintenance of PCEP sessions with a large number of PCCs to be decoupled from path computation, which improves the scalability of the PCE architecture. In the scope of this document, the term 'computation loop' is used to describe the behavior of the PCEP message exchange looping between the PCC and the PCE or between PCEs, resulting in frequent path calculations, path reporting and path updates to the network, resulting in constant load on the PCE and oscillation of data plane traffic after each subsequent path update. This section contains illustrative examples to showcase the need for inter-PCE stateful PCEP sessions; further examples are provided in , , and .

Reporting LSP Changes When using a stateful PCE (), a PCC can synchronize LSP state information to the stateful PCE. If the PCC grants control of the LSP to the PCE (called delegation ), the PCE can update the LSP parameters at any time. In a multi-PCE deployment (redundancy, load-balancing...), with the specification defined in , when a PCE makes an update, the PCC is responsible for reporting the LSP parameter updates to all PCEs. This convergence delay may affect the reaction time of the other PCEs if they need to take action after being notified of the LSP parameter change. Apart from the synchronization from the PCC, it is also useful if there is a synchronization mechanism between the stateful PCEs. As a stateful PCE makes changes to its delegated LSPs, these changes (pending LSPs and the sticky resources ) can be synchronized to the other PCEs.

Active and Standby PCEs In , PCE1 is responsible for computing paths for the PCC, and PCE2 is on standby. When there is a change in LSP1, the PCC reports it to PCE1. In addition, PCE1 synchronizes to PCE2 the intended state of LSP1, including pending LSP state and the sticky resources accounted for during path computation. This gives PCE2 visibility of state that the PCC has not yet confirmed, or has not yet been asked to apply, which a report from the PCC alone cannot provide at that point in time. PCE2 therefore has a more accurate view for its own computations and for taking over from PCE1.

Split-Brain In a resiliency case, a PCC has redundant PCEP sessions toward multiple PCEs. In such a case, a PCC gives control of an LSP to a single PCE only, and only this PCE is responsible for the path computation for the delegated LSP: the PCC achieves this by setting the D flag only toward the active PCE selected for delegation. The election of the active PCE to delegate an LSP is controlled by each PCC. The PCC usually elects the active PCE by a locally configured policy (by setting a priority). Upon PCEP session failure, or active PCE failure, the PCC may decide to elect a new active PCE by sending a new Path Computation LSP State Report (PCRpt) message with the D flag set to this new active PCE. When the failed PCE or PCEP session comes back online, it will be up to the PCC implementation whether to revert back to the original PCE. Any change of the PCE that computes the path, whether the initial failover or a later reversion, may lead to some disruption of the existing path if the computation results of the two PCEs are not exactly the same. By considering a network with multiple PCCs and implementing multiple stateful PCEs for redundancy purposes, it is not likely that all PCCs delegate their LSPs to the same PCE.

Two PCEs with Shared Responsibility In the example in , PCC1 and PCC2 are configured to delegate their respective LSPs, LSP1 and LSP2, to PCE1. Therefore, PCE1 is responsible for computing paths for both LSPs. If the PCEP session between PCC2 and PCE1 fails, PCC2 delegates LSP2 to PCE2. PCE1 then remains responsible only for the path computation of LSP1, while PCE2 is responsible for the path computation of LSP2. Once the PCC2-PCE1 session is restored, PCC2 continues using PCE2 as the active PCE, assuming no preemption in this example. The result is a persistent situation where each PCE is responsible for a subset of path computations. This situation is called a split-brain scenario, where multiple PCEs perform computation independently in a deployment or use case that requires coordinated computation across a set of associated LSPs. Such lack of coordination is particularly problematic when one LSP path computation depends on another, for example for path disjointness (see ) or bidirectional LSPs (see ). The set of LSPs that are dependent on each other may start from different head-ends.

Managing Link-Disjoint LSPs | PCC2 | | | | | <---------------------- | | | | +------+ +------+ | | | | | | +------+ +------+ | | | PCC3 | ----------------------> | PCC4 | | | | | <---------------------- | | | | +------+ +------+ | | | \ / \_________________________________________/ _________________________________________ / \ / +------+ +------+ \ | | PCE1 | | PCE2 | | | +------+ +------+ | | | | +------+ 10 +------+ | | | PCC1 | ----- R1 ---- R2 ------- | PCC2 | | | +------+ | | +------+ | | | | | | | | | | +------+ | | +------+ | | | PCC3 | ----- R3 ---- R4 ------- | PCC4 | | | +------+ +------+ | | | \ / \_________________________________________/ ]]> In , the requirement is to create two link-disjoint LSPs: PCC1->PCC2 and PCC3->PCC4. In the topology, all link cost metrics are set to 1 except for the link 'R1-R2', which has a metric of 10. The PCEs are responsible for the path computation, and PCE1 is the active PCE for all PCCs in the nominal case. provides several scenarios for illustrative purposes, and shows how the procedures of this document apply to them. There are many other cases where the solutions defined in this document are also applicable.

Scalability If the number of PCCs is really high, it may be too resource-consuming for a single PCE instance to maintain all the PCEP sessions while at the same time performing all path computations. Decoupling PCEP session maintenance and computation across a group of PCEs that share LSP state allows increasing scaling of the PCE architecture. gives an example.

Relationship to H-PCE describes general considerations and use cases for the deployment of Stateful PCE(s) using the Hierarchical PCE architecture. In this architecture, there is a clear need to communicate between a child stateful PCE and a parent stateful PCE. The procedures in this document address the synchronization of state between peer stateful PCEs and do not require deployment of a parent PCE. The use of these state-sync procedures within an H-PCE deployment is outside the scope of this document.

Solution Overview The solution uses the following mechanisms, described in :

The state-sync session (, ): a PCEP session between stateful PCEs, with a new capability flag. Each PCE acts as both PCE and PCC to synchronise state.
The state-sync PCE set: the PCEs that share state for a given set of PCCs, LSPs, or association groups, which MUST be connected in a full mesh of state-sync sessions ().
The forwarding rules (, ): a PCE forwards to its state-sync peers only the PCRpt messages it receives from a PCC, never a PCRpt received from a PCE; a PCUpd from the computing PCE is sent to all state-sync peers, but only the delegated PCE forwards it to the PCC. These rules prevent loops ().
The computation priority and sub-delegation (, ): one PCE computes for each LSP, PCC, or association group. The other PCEs are backups in priority order. If the delegated PCE is not the computing PCE, it sub-delegates the LSP to that PCE using the D flag.
A source list per LSP () and a PCC information database (), maintained in addition to the LSP-DB.
The ORIGINAL-LSP-DB-VERSION TLV () and the optional PCEP-PATH-VECTOR TLV ().

The solution builds upon the protocol extensions for stateful PCE in , synchronization optimizations in , and PCE-initiation in .

State-Sync Session This document specifies a mechanism to set up a PCEP session between the stateful PCEs. Creating a PCEP session between PCEs is already enabled for multiple scenarios like the one described in (multiple PCEs that are handling part of a path computation) and (hierarchical PCE). However, that earlier work focused only on the sessions between stateless PCEs. Stateful PCE brings additional features (LSP state synchronization, path update, delegation, ...). Thus, some new behaviors need to be defined on the inter-PCE PCEP session. This inter-PCE PCEP session allows the exchange of LSP state between PCEs and is called a "state-sync session" in this document. For example, in the scenario in , there is no possibility to compute disjointness as there is no PCE that is aware of both LSPs.

Partitioned Visibility Amongst PCEs PCC2 +----------+ / D=1 / +---------+ +---------+ | PCE1 | | PCE2 | +---------+ +---------+ / D=1 / +----------+ | PCC3 | LSP: PCC3->PCC4 +----------+ ]]> If we add a state-sync session as shown in , PCE1 will synchronize its LSP state to PCE2 via PCRpt messages; PCE2 will similarly synchronize its state to PCE1. All the PCEs will be aware of all LSPs, even if a PCC->PCE session is down. PCEs will then be able to compute disjoint paths.

Partitioned Visibility With State Synchronization PCC2 +----------+ / D=1 / +---------+ PCEP +---------+ | PCE1 | ----- | PCE2 | +---------+ +---------+ / D=1 / +----------+ | PCC3 | LSP : PCC3->PCC4 +----------+ ]]> The procedures associated with this state-sync session are defined in . By just adding this state-sync session, it does not ensure that a path with LSP association-based constraints can always be computed and does not prevent the computation loop, but it increases resiliency and ensures that PCEs will have the state information for all LSPs. Also, this session will allow a PCE to update the other PCEs, providing a faster synchronization mechanism than relying on PCCs only.

Computation Priority Relationship between PCEs As seen in and , performing a path computation in a split-brain scenario (multiple PCEs responsible for computation) may provide a non-optimal LSP placement, no path, or computation loops. To achieve better efficiency, an LSP association constraint-based computation may require that a single PCE performs the path computation for all LSPs in the association group. Note that it could be all LSPs belonging to a particular association group, or all LSPs from a particular PCC, or all LSPs in the network that need to be delegated to a single PCE based on the deployment scenarios. This document specifies a computation priority between the PCEs of a state-sync PCE set, from which a single computing PCE is determined, with the other PCEs acting as backups in priority order. Using this priority, PCEs agree on the PCE that will be responsible for the computation for a particular association group or set of LSPs. The priority can be set per association group, per PCC, or globally for all LSPs. The rest of the text considers the association group as an example. When a single PCE is performing the computation for a particular association group, no computation loop can happen, and an optimal placement will be provided. The other PCEs will only act as state collectors and forwarders. In the scenario described in , PCE1 and PCE2 will decide that PCE1 will be responsible for the path computation of both LSPs. If we first configure PCC1->PCC2, PCE1 computes the shortest path as it is the only LSP in the disjoint-group that it is aware of: R1->R3->R4->R2->PCC2 (shortest path). When PCC3->PCC4 is configured, PCE2 will not perform computation even if it has delegation, but forwards the delegation via PCRpt message to PCE1 through the state-sync session. PCE1 will then perform disjointness computation and will move PCC1->PCC2 onto R1->R2->PCC2 and provides an ERO to PCE2 for PCC3->PCC4: R3->R4->PCC4. The PCE2 will further update the PCC3 with the new path.

Procedures and Protocol Extensions

Opening a State-Sync Session and Capability Advertisement A PCE indicates its support of state-sync procedures during the PCEP Initialization phase . The OPEN object in the Open message MUST contain the "Stateful PCE Capability" TLV defined in . A new P (INTER-PCE-CAPABILITY) flag is introduced to indicate the support of state-sync. This document adds a new bit in the Flags field with:

P (INTER-PCE-CAPABILITY - 1 bit - TBD4): If set to 1 by a PCE, the PCE indicates that the session MUST follow the state-sync procedures as described in this document. If the P bit is set by both PCEP speakers on the session, the procedures MUST be used. If a PCEP speaker receives a STATEFUL-PCE-CAPABILITY TLV with P=0 while it advertised P=1, or if both PCEP speakers set the P flag to 0, the session is set up, but the state-sync procedures MUST NOT be applied on this session. A PCE MAY decide to close a session if the received setting of the P flag is not acceptable.

The U flag MUST be set when sending the STATEFUL-PCE-CAPABILITY TLV with the P flag set. In case the U flag is not set along with the P flag, the state-sync capability is not enabled and it is considered as if the P flag is not set. The procedures of this document depend on the LSP-DB-VERSION TLV and the SPEAKER-ENTITY-ID TLV defined in : a PCE that sets the P flag MUST support . The S flag SHOULD be set together with the P flag so that the optimized synchronization procedures of can be used on the state-sync session; this reduces the amount of state exchanged when a state-sync session is re-established. A full synchronization remains a valid behavior when the S flag is not set. The TLVs and the Notification-type defined in this document MUST NOT be sent on a session on which the state-sync capability has not been negotiated, and MUST be ignored if received on such a session. PCRpt and PCUpd messages received on such a session are processed according to only and MUST NOT be forwarded to any state-sync session. State synchronization procedures are independent of support for other PCEP capabilities, for example the Path Setup Type capability defined in . While two PCEs MUST support state-sync extensions to perform synchronization, they do not need to support a symmetrical set of additional capabilities. In cases of asymmetry, state-sync SHOULD continue; however, the PCE MUST handle information and behavior according to the specific rules defined for those capabilities, such as ignoring paths with an unsupported Path Setup Type or reporting an error as defined in the relevant capability specification. A PCE that does not support a capability needed for a set of LSPs cannot act as computing PCE for those LSPs; see .

State Synchronization When the state-sync capability has been negotiated between stateful PCEs, each PCEP speaker will behave as a PCE and as a PCC at the same time regarding the state synchronization as defined in . This means that each PCEP Speaker:

MUST send a PCRpt message toward its neighbor with the S flag set for each LSP in its LSP database learned from a PCC. (PCC role)
MUST send the End Of Synchronization Marker toward its neighbor when all LSPs have been reported. (PCC role)
MUST wait for the LSP synchronization from its neighbor to end (receiving an End Of Synchronization Marker). (PCE role)

The process of synchronization runs in parallel on each PCE (with no defined order). Before the LSP state synchronization starts, the PCEs exchange the information of the PCCs connected to them as described in . The optimized state synchronization procedures of SHOULD be used when the S flag has been set by both PCEs; see . When a PCEP Speaker sends a PCRpt on a state-sync session, it MUST add the SPEAKER-ENTITY-ID TLV (defined in ) in the LSP Object, the value used will refer to the 'owner' PCC of the LSP. If a PCEP Speaker receives a PCRpt on a state-sync session without this TLV, it MUST discard the PCRpt message and it MUST reply with a PCErr message using error-type=6 (Mandatory Object missing) and error-value=TBD1 (SPEAKER-ENTITY-ID TLV missing). Errors during the synchronization are handled as specified in Section 5.6 of and in , which result in the session being closed. If the state-sync session fails before the End Of Synchronization Marker has been received, the state already received on that session is handled as for any state-sync session failure ().

Incremental Updates and Report Forwarding Rules During the life of an LSP, its state may change (path, constraints, operational state...), and a PCC will advertise a new PCRpt to the PCE for each such change. When propagating LSP state changes from a PCE to other PCEs, the sending PCE MUST propagate only the freshest state received from the PCC, as determined by the LSP-DB version semantics (see ). When a PCE receives a new PCRpt from a PCC with the LSP-DB-VERSION TLV (defined in ), the PCE MUST forward the PCRpt to all its state-sync sessions and MUST add the appropriate SPEAKER-ENTITY-ID TLV in the PCRpt. In addition, it MUST add a new ORIGINAL-LSP-DB-VERSION TLV (described below). The ORIGINAL-LSP-DB-VERSION contains the LSP-DB-VERSION coming from the PCC. When a PCE receives a new PCRpt from a PCC without the LSP-DB-VERSION TLV, it MUST NOT forward the PCRpt on any state-sync sessions and MUST log such an event on the first occurrence for that PCC on the current PCEP session (so that the condition is visible to the operator without flooding the log). When a PCE receives a new PCRpt from a PCC with the R flag (Remove) set and an LSP-DB-VERSION TLV, the PCE MUST forward the PCRpt to all its state-sync sessions keeping the R flag set (Remove) and MUST add the appropriate SPEAKER-ENTITY-ID TLV and ORIGINAL-LSP-DB-VERSION TLV in the PCRpt message. When a PCE receives a PCRpt from a state-sync session, it MUST NOT forward the PCRpt to other state-sync sessions. This helps to prevent message loops between PCEs (see ). As a consequence, a full mesh of state-sync sessions between the PCEs of a state-sync PCE set is REQUIRED (see and ). When a PCRpt is forwarded, all the original objects and values are kept. As an example, the PLSP-ID used in the forwarded PCRpt will be the same as the original one used by the PCC. Thus, an implementation supporting this document MUST consider SPEAKER-ENTITY-ID TLV and PLSP-ID together to uniquely identify an LSP on the state-sync session. The ORIGINAL-LSP-DB-VERSION TLV is encoded as shown in and MUST always contain the LSP-DB-VERSION received from the owner PCC of the LSP. The Length field is set to 8 because the version value is encoded as a 64-bit quantity (8 octets).

The ORIGINAL-LSP-DB-VERSION TLV Using the ORIGINAL-LSP-DB-VERSION TLV allows a PCE to keep using optimized synchronization () with another PCE. In such a case, the PCE will send a PCRpt to another PCE with both ORIGINAL-LSP-DB-VERSION TLV and LSP-DB-VERSION TLV. The ORIGINAL-LSP-DB-VERSION TLV will contain the version number as allocated by the PCC, while the LSP-DB-VERSION will contain the version number allocated by the local PCE.

Maintaining LSP States from Different Sources When a PCE receives a PCRpt on a state-sync session, it stores the LSP information into the original PCC address context (as the LSP belongs to the PCC). A PCE MUST maintain a single state for a particular LSP and MUST maintain the list of sources (the source list) from which it learned that state. A PCEP speaker may receive state information for a particular LSP from different sources: the PCC that owns the LSP (through a regular PCEP session) and some PCEs (through PCEP state-sync sessions). The freshness of the state is determined by the version number allocated by the owner PCC: the LSP-DB-VERSION when the state is received from the PCC, and the ORIGINAL-LSP-DB-VERSION when the state is received from a PCE. A received state is "fresher" than the stored state when its version number is greater than the stored version number, taking wrap-around into account. A PCEP speaker MUST always keep the freshest state in its LSP database, overriding the previously received information. A PCE, receiving a PCRpt from a PCC, updates the state of the LSP in its LSP-DB with the newly received information. When receiving a PCRpt from another PCE, a PCE updates the LSP state only if the ORIGINAL-LSP-DB-VERSION present in the PCRpt indicates it is fresher than the current ORIGINAL-LSP-DB-VERSION of the stored LSP state. This ensures that a PCE never tries to update its stored LSP state with old information. Each time a PCE updates an LSP state in its LSP-DB, it MUST clear the source list associated with the LSP state and MUST re-initialize it with the address of the speaker from which the fresher state was received. When a PCE receives a PCRpt that has an ORIGINAL-LSP-DB-VERSION (if coming from a PCE) or an LSP-DB-VERSION (if coming from the PCC) equal to the current ORIGINAL-LSP-DB-VERSION of the stored LSP state, it MUST add the source speaker address to the source list, if not already present. When a PCE receives a PCRpt with the R flag set (LSP deletion) from a particular source, it MUST remove this particular source from the source list associated with this LSP. When the source list becomes empty as a result of PCRpt messages with the R flag set, the LSP state MUST be removed. This means that all the sources currently associated with the stored state must send a PCRpt with R=1 for an LSP to make the PCE remove the LSP state. The replacement of a stored state by a fresher one re-initializes the source list but is not a deletion, and a source list that becomes empty because of a state-sync session failure does not remove the LSP state either ().

Computation Priority between PCEs and Sub-delegation A computation priority is necessary to ensure that a single PCE will perform the computation for all the LSPs in an association group: this will allow for a more optimized LSP placement and will prevent computation loops. The operator MUST provision, on every PCE of a state-sync PCE set, the computation priority of every PCE of the set, including the local PCE, so that all the PCEs that handle LSPs in a common LSP association group are aware of each other and of each other's computation priority; see and . Note that there is no need for the PCC to be aware of this. The computation priority is a number ranging from 0 to 7, where 0 represents the lowest priority, while 7 represents the highest. The PCE having the highest priority MUST be responsible for the computation. If several PCEs have the same priority value, their IP address MUST be used as a tie-breaker to provide a rank: the highest IP address has more priority. An IPv4 address MUST be converted to its IPv4-mapped IPv6 form before comparison, per . The computation priorities are configured through local operator policy on each PCE. The priority for local and remote PCEs can be set at the global level, so that the highest priority PCE handles all path computations, or at a more granular level (per association group or per PCC), so that a PCE has the highest priority for only a subset of LSPs or association groups. The most specific configured level applies; see for the precedence rules. If a PCE has no computation priority configured for a remote PCE at any level, it MUST treat that PCE as having priority 0. In future, PCEs could also advertise and discover these parameters via PCEP; those details are out of the scope of this document and left for future specification (see ). A PCEP Speaker receiving a PCRpt from a PCC with the D flag set that does not have the highest computation priority MUST forward the PCRpt on all state-sync sessions (as per ) and MUST set the D flag on the state-sync session toward the highest priority PCE; the D flag MUST be unset on all other state-sync sessions. This behavior is similar to the delegation behavior handled at the PCC side and is called a sub-delegation (the PCE sub-delegates the control of the LSP to another PCE). When a PCEP Speaker sub-delegates an LSP to another PCE, it loses control of the LSP and MUST NOT update it by its own decision. When a PCE receives a PCRpt with the D flag set on a state-sync session, it is granted control over the LSP as a regular PCE would be. A PCE that has been granted control by sub-delegation MUST NOT sub-delegate the LSP further, even if, from its own view, another PCE has a higher computation priority: the view of the delegated PCE governs the LSPs it received from its PCCs. Sub-delegation is therefore never deeper than one level. Sub-delegation is not visible to the PCC. The PCC continues to interact only with the PCE to which it delegated the LSP: it receives PCUpd messages only from that PCE and sends its PCRpt messages to it. The sub-delegated PCE never acts toward the PCC directly; it may not even have a PCEP session with that PCC. When a PCE holds the delegation for an LSP directly from the PCC, is the highest priority PCE, and needs to update this LSP, it MUST send a PCUpd message to all state-sync sessions and to the PCC session on which it received the delegation. The D flag MUST be unset in the PCUpd sent on the state-sync sessions and MUST be set in the PCUpd sent to the PCC. In the case of sub-delegation, the computing PCE MUST send the PCUpd to all state-sync sessions (it has no direct delegation from a PCC). The D flag MUST be set in the PCUpd sent on the state-sync session to the PCE that sub-delegated this LSP and MUST be unset in the PCUpd sent on the other state-sync sessions. Sending the PCUpd to all state-sync sessions keeps all PCEs aware of the path updates that have been sent toward the PCC but not yet confirmed by a corresponding PCRpt, and avoids transient state divergence during re-optimization. The PCUpd sent over a state-sync session MUST contain the SPEAKER-ENTITY-ID TLV in the LSP Object (the value used MUST identify the target PCC). The PLSP-ID used is the original PLSP-ID generated by the PCC and learned from the forwarded PCRpt. If a PCE receives a PCUpd on a state-sync session without the SPEAKER-ENTITY-ID TLV, it MUST discard the PCUpd and MUST reply with a PCErr message using error-type=6 (Mandatory Object missing) and error-value=TBD1 (SPEAKER-ENTITY-ID TLV missing). A receiving PCE that currently holds delegation for that LSP on an active PCEP session to the PCC identified by SPEAKER-ENTITY-ID MUST forward the PCUpd to that PCC and MUST remove the SPEAKER-ENTITY-ID TLV from the LSP Object. A receiving PCE that does not currently hold delegation for that LSP on an active PCEP session to that PCC MUST NOT forward the PCUpd to the PCC; it keeps the update only for state tracking and for subsequent synchronization logic. The PCUpd is acknowledged through the PCRpt originated by the PCC. When the PCC receives the PCUpd from the delegated PCE, it acknowledges it with a PCRpt as per . On receiving this PCRpt, the delegated PCE applies the forwarding rules defined in on the state-sync sessions, so the acknowledgement is relayed to the computing PCE. The PCRpt received by a PCE that did not originate the PCUpd is not required to carry the Stateful PCE Request Parameters (SRP) object , and any SRP-ID-number it carries is scoped to that PCEP session (per ). Therefore, a PCE MUST NOT rely on the SRP object to correlate a PCRpt received on a state-sync session with a PCUpd it sent. The computing PCE MUST correlate the relayed PCRpt with its PCUpd using the identity of the LSP, that is, the SPEAKER-ENTITY-ID TLV together with the PLSP-ID, which are preserved end to end by the forwarding rules. The ORIGINAL-LSP-DB-VERSION TLV establishes that the relayed PCRpt is fresher than the state the PCUpd was computed from. The content of the PCRpt (including any LSP-ERROR-CODE TLV ) tells the computing PCE whether its update was applied or rejected. The delegated PCE handles the SRP object on its session with the PCC as per .

State-Sync Session Failure, Switch-over, and Reversion A failure may be detected through loss of reachability to the peer, state-sync session failure, or a local policy that marks the PCE as temporarily unusable. The trigger and threshold are determined by local policy. The Redelegation Timeout and State Timeout procedures of do not apply to state-sync sessions. On failure of a state-sync session, the local PCE:

MUST remove the failed peer from the source list of every LSP state in which it appears. If the source list becomes empty, the LSP state MUST be retained and marked as stale until refreshed or removed by the procedures of ; it MUST NOT be removed because of the failure. Stale state SHOULD continue to be used for path computation.
MUST remove the PCC information entries whose only source was the failed peer (). LSP state is retained even when no PCC information entry remains for its PCC.
MUST consider any sub-delegation received from the failed peer as revoked, and MUST NOT send further PCUpd messages for those LSPs.
MUST re-evaluate the sub-delegation of every LSP it had sub-delegated to the failed peer, using the same priority policy for the reachable state-sync peers, and either sub-delegate the LSP to the next highest priority PCE (D flag set in the PCRpt sent to that PCE, unset towards the others) or, if it is itself the highest priority reachable PCE, perform the computation itself.

No re-synchronisation with the other peers is triggered by the failure of a single state-sync session, as the same LSP state is normally still held, with the owner PCC and the other peers remaining as sources. If the PCC detects failure of the PCE to which it delegated the LSP, the PCC uses the existing delegation procedures of , and the newly delegated PCE applies . The PCC does not need to know the PCE priority ranking. A PCE that receives a sub-delegation MUST accept it and act as computing PCE for that LSP even if, from its own view, a PCE with a higher computation priority is reachable. The sub-delegating PCE uses its own view for the LSPs received from its PCCs; the receiving PCE cannot sub-delegate them again. When the PCEs of a set have different views of which peers are reachable, different LSPs of an association group may therefore temporarily have different computing PCEs. This gives less optimal results for association-based computation () but cannot create a message loop or a computation loop, because the forwarding rules and the limit of one sub-delegation still apply. Such a condition SHOULD be reported to the operator (). When a state-sync session to a higher priority PCE is re-established, the state synchronisation of MUST complete before any sub-delegation towards that PCE. This means that the End Of Synchronization Marker has been received from that PCE. The delegated PCE MAY then revert the sub-delegation to it (D flag set in the PCRpt sent to the returning PCE, unset towards the previous computing PCE). Local policy determines whether to revert. As with the initial switch-over, reversion may disrupt the existing path if the two PCEs compute different results ().

Information Received via Open Message from PCC To ensure uniform information across all PCEs, each PCE needs to relay the information it receives from the PCCs in the Open message to other PCEs via the state-sync session. This includes various PCC capabilities and parameters such as Maximum Segment Identifier (SID) Depth (MSD). Each PCE records this information, together with the source(s) it was learned from (the direct session with the PCC and/or state-sync peers), in a PCC information database. This database is used, in particular, to select a peer with a session to a given PCC (). As per , the PCEP Notification message (PCNtf) can be sent by a PCEP speaker to notify its peer of a specific event. A PCE MUST notify the other state-sync PCEs of the information it receives from the PCC's open message. Section 7.14 of specifies the NOTIFICATION object. This document adds a new Notification-type=TBD6 (Inter-PCE State-sync) and two Notification-values (Notification-value=1 (Add PCC's Open Information) and Notification-value=2 (Remove PCC's Open Information)). For Notification-type=TBD6, the NOTIFICATION object encodes the SPEAKER-ENTITY-ID TLV (with values that identify the PCC) and any other TLV that can be carried inside the OPEN object as a way to signal the PCC's information it received via the open message to other state-sync PCEs.

Notification-value=1: Add PCC's Open Information. On session establishment with a PCC, a PCE with state-sync capability MUST send this notification to other state-sync PCEs with the SPEAKER-ENTITY-ID TLV with values that identify the PCC and any other TLVs encoded in the OPEN object received from the PCC. On session establishment with a state-sync PCE, the PCE MUST also send this notification for each of the PCCs with which it currently has a session established. The PCE MUST send these notifications prior to the State Synchronization (described in ). Note that the PCNtf can be used to carry multiple NOTIFICATION objects, one for each PCC. On receiving this notification, the PCE adds the information to its PCC information database and records the sending peer as a source.
Notification-value=2: Remove PCC's Open Information. When a PCE with state-sync capability clears the state of a PCC whose session has gone down, that is, on expiry of the State Timeout Interval of , it MUST send this notification to other state-sync PCEs with the SPEAKER-ENTITY-ID TLV with values that identify the PCC. It is not sent on session down while the PCE still retains the PCC's state. On receiving this notification, the PCE removes the sending peer from the sources of the entry, and removes the entry when no source is left.

A PCE may receive this Notification from multiple PCEs that a given PCC has a session with, and uses the recorded sources in a similar way to the source list of . In case of the termination of a state-sync session, the entries whose only source was that peer are removed (); upon re-establishment, the Notification-value=1 exchange rebuilds them before LSP state synchronization starts.

Association Group All LSPs belonging to the same association group MUST be configured with the same computation priorities for the PCEs. A PCE only computes a path using an association-group constraint if it has control of all of the LSPs in the association group, either by delegation from the PCC or by sub-delegation. Otherwise, an implementation MAY use a local policy on the PCE to decide whether it does not compute a path at all for this set of LSPs, or whether it computes a path by relaxing the association-group constraint. The same local policy applies if the PCE does not control all LSPs in the group because the priorities for an association group are not configured consistently on all PCEs, because a state-sync session is down (), or because a newly added PCE has not yet been provisioned (and is therefore treated as priority 0, ). No LSP state is removed in any of these cases.

Passive Stateful Procedures In the passive stateful PCE architecture , the PCC is responsible for triggering a path computation request using a PCReq message to its PCE. There is no delegation, and hence no sub-delegation, in this mode; only the request is forwarded. If a PCE receives a PCReq for an LSP and finds that it does not have the highest computation priority for this LSP, or group, it MUST forward the PCReq message to the highest priority PCE over the state-sync session. When the highest priority PCE receives the PCReq, it computes the path and generates a Path Computation Reply (PCRep) message toward the PCE that forwarded the request. This PCE MUST then forward the PCRep to the requesting PCC on the session on which the PCReq was received. The handling of the LSP object and the SPEAKER-ENTITY-ID TLV in PCReq and PCRep is similar to PCRpt/PCUpd messages. If a PCC sends a PCReq for the same LSP to several PCEs, each of them forwards it in this way and the PCC receives one PCRep per PCReq, all computed by the same PCE.

PCE Initiation Procedures A PCE may not have a direct PCEP session to a PCC to initiate an LSP as per . A PCE MAY send the Path Computation LSP Initiate Request (PCInitiate) message on a state-sync session to another PCE to request it to create a PCE-initiated LSP on its behalf. The requesting PCE MUST select a peer that, according to its PCC information database (), has a session with the head-end PCC. The PCInitiate is sent only to the selected peer. If the selected PCE is able to initiate the LSP, it reports it on the state-sync session via a PCRpt message, and the resulting LSP state reaches all the PCEs of the set through the normal PCRpt forwarding rules (). If the selected PCE does not have a session to the head-end, for example because the PCC information was stale or because the session went down in the meantime, it MUST send a PCErr message with Error-type=24 (LSP instantiation error) and Error-value=TBD5 (No PCEP session with the headend). The requesting PCE MAY then retry via another available state-sync PCE that has a session with the head-end.

Loop Prevention and Loop Detection The forwarding rules in this document prevent message loops between PCEs: a PCRpt or PCUpd received on a state-sync session is never forwarded to another state-sync session (, ), a PCE that has received a sub-delegation never sub-delegates further (), and the PCEs of a state-sync PCE set are connected in a full mesh. Following these rules, each PCEP message is relayed at most once between PCEs and cannot loop. Computation loops are prevented by the computation priority, which ensures that a single PCE computes for all the LSPs of an association group. The optional PCEP-PATH-VECTOR TLV defined below is used for diagnosis rather than loop prevention. It records the PCEP speakers through which a message has passed, allowing a speaker to detect a message returning to it following a misconfiguration or an implementation error. Other uses, such as recording hop-by-hop information, are outside the scope of this document.

PCEP-PATH-VECTOR TLV This document introduces the PCEP-PATH-VECTOR TLV (type TBD3) to be encoded in the LSP Object with the format shown in .

The PCEP-PATH-VECTOR TLV The TLV format and padding rules are as per . The PCEP-SPEAKER-INFORMATION field has the format shown in .

The PCEP-SPEAKER-INFORMATION

Length: defines the total length of the PCEP-SPEAKER-INFORMATION field.
ID Length: defines the length of the Speaker Entity identity field, not counting any padding.
Speaker Entity identity: same as the value portion of the SPEAKER-ENTITY-ID TLV. Padded with trailing zeros to a 4-byte boundary.
The PCEP-SPEAKER-INFORMATION may also carry some optional sub-TLVs, so each PCEP speaker can add local information that could be recorded. This document does not define any sub-TLVs.

The PCEP-PATH-VECTOR TLV MAY be carried in the LSP Object. Its usage is purely optional; its absence does not affect loop prevention. The list of speakers within the PCEP-PATH-VECTOR TLV MUST be ordered. When sending a PCEP message (PCRpt, PCUpd, or PCInitiate), a PCEP Speaker MAY add the PCEP-PATH-VECTOR TLV with a PCEP-SPEAKER-INFORMATION containing its own information. If the PCEP message sent is the result of a previously received PCEP message, and if the PCEP-PATH-VECTOR TLV was already present in the initial message, the PCEP speaker SHOULD append a new PCEP-SPEAKER-INFORMATION containing its own information at the end of the TLV. If a PCEP speaker receives a message with a PCEP-PATH-VECTOR TLV and finds its speaker information already present in the PCEP-PATH-VECTOR TLV, this represents a message loop: it MUST ignore the PCEP message and SHOULD log it as an error (). This check applies whenever the TLV is present.

Security Considerations The security considerations described in , , , and apply to the extensions described in this document as well. Additional considerations related to state synchronization and sub-delegation between stateful PCEs are introduced, as they could be spoofed and could be used as an attack vector. An attacker could attempt to create too much state in an attempt to load the PCEP peer. The PCEP peer could respond with a PCErr message as described in . An attacker could impact LSP operations by creating a bogus state. Further, state synchronization between stateful PCEs could provide an adversary with the opportunity to eavesdrop on the network. A compromised or misbehaving PCE can cause more harm than on an isolated PCC-PCE session. Bogus state injected on one state-sync session is propagated throughout the state-sync PCE set. Through sub-delegation, it can also affect LSPs of PCCs that have no direct session with that PCE. Securing both state-sync and PCC-PCE sessions using Transport Layer Security (TLS) with mutual authentication, as per the recommendations and best current practices in , is RECOMMENDED. State-sync sessions SHOULD be accepted only from provisioned peers.

[Note to the RFC Editor - remove this section before publication, as well as remove the reference to RFC 7942.] This section records the status of known implementations of the protocol defined by this specification at the time of posting of this Internet-Draft, and is based on a proposal described in . The description of implementations in this section is intended to assist the IETF in its decision processes in progressing drafts to RFCs. Please note that the listing of any individual implementation here does not imply endorsement by the IETF. Furthermore, no effort has been spent to verify the information presented here that was supplied by IETF contributors. This is not intended as, and must not be construed to be, a catalog of available implementations or their features. Readers are advised to note that other implementations may exist. According to , "this will allow reviewers and working groups to assign due consideration to documents that have the benefit of running code, which may serve as evidence of valuable experimentation and feedback that have made the implemented protocols more mature. It is up to the individual working groups to use this information as they see fit". At the time of posting this document, there are no known implementations of this mechanism. It is believed that some vendors are considering implementations, but these plans are too vague to make any further assertions.

Operational Considerations All manageability requirements and considerations listed in , , , and apply to the PCEP protocol extensions defined in this document. In addition, the requirements and considerations listed in this section apply.

Deployment Design Guidelines The PCEs that share state for a given set of PCCs, LSPs, or association groups MUST be connected by state-sync sessions in a full mesh (, ). The full mesh requirement applies to the participating state-sync PCE set, not necessarily to every PCE in an operator network. Operators can size each state-sync set according to the LSP associations, PCCs, and redundancy policy for which state must be shared. Two or three PCEs are typical for redundancy. Load sharing () may need more, with per-association or per-PCC priorities. For association-based computation, the set needs to include every PCE that can receive delegation for an LSP in the group. Operators SHOULD use the smallest set that meets the objective (). The computation priorities MUST be provisioned consistently on all PCEs of the set, at every configuration level in use (). Inconsistent provisioning does not create loops (), but it prevents association-based computation and can cause an LSP to be sub-delegated to a PCE that does not expect it. A PCE added to the set is treated as priority 0 until provisioned (), so it can be introduced without disruption. The provisioned priorities determine how computation is shared between the PCEs.

An operator MUST be allowed to configure the capability to support state-sync procedures for an inter-PCE session. They MUST be allowed to configure a computation priority of the local and remote PCEs at the global level. They SHOULD also be allowed to configure the computation priority of the local and remote PCEs per association (or a range of them). Further, they SHOULD also be allowed to configure computation priority per PCC (or a range of them). An implementation MAY support other such configuration levels for computation priority of the local and remote PCEs. The association-group priority takes precedence over the PCC priority, and the PCC priority takes precedence over the global priority. The most specific configured level therefore applies. The same precedence rules MUST apply on all the PCEs of the set. Where the PCEs of a set do not support the same capabilities (), the operator should assign the priorities so that a PCE is never the highest priority PCE for LSPs it cannot compute.

The PCEP YANG module does not include support for state-sync. It can be extended to include, at the very least:

Indicate whether state-sync is supported at the local and remote peer
Enablement and disablement of state-sync capability
Indicate whether the PCEP session between PCEs has state-sync enabled
View the sources via which LSP information is received
View the PCC information database, including PCCs learned via state-sync peers
Setting computation priority as per
View which PCE is the computing PCE for an LSP and any sub-delegation

Mechanisms defined in this document do not imply any new liveness detection and monitoring requirements in addition to those already listed in and .

In addition to the verification requirements listed in , an operator MUST be able to verify that state-sync sessions are established to all members of the state-sync PCE set (the full mesh) and that the computation priorities are provisioned consistently across the members. Priority consistency needs to be checked through the management system, as priorities are not exchanged in PCEP. An implementation SHOULD raise an alarm when a configured state-sync session is down, when a sub-delegation is received for an LSP for which the local PCE does not consider itself the computing PCE (), and when a message is ignored because of the PCEP-PATH-VECTOR TLV check ().

Mechanisms defined in this document do not imply any new requirements on other protocols.

Mechanisms defined in this document improve the network operations by alleviating the problems described in . The "full mesh" requirement applies only among PCEs that participate in inter-PCE state synchronization for the same set of PCCs or associations. In operational deployments, this typically involves a small number of PCEs (e.g., two or three for redundancy), making a full mesh feasible for deterministic state consistency and loop prevention; see for other use cases.

Acknowledgements Thanks to for urging better use of terms. Thanks to Andrew Stone for shepherding this document and for providing useful reviews and comments. Thanks to Ketan Talaulikar for the detailed AD review.

IANA Considerations This document requests IANA actions to allocate code points for the protocol elements defined in this document. All the registries referred to in this section are part of the "Path Computation Element Protocol (PCEP) Numbers" registry group (https://www.iana.org/assignments/pcep/).

PCEP-Error Object This document defines a new Error-Value within the "Mandatory Object Missing" Error-Type and a new Error-Value within the "LSP instantiation error" Error-Type. IANA is requested to allocate the new error values within the "PCEP-ERROR Object Error Types and Values" registry, as follows:

Error-Type	Meaning	Reference
6	Mandatory Object Missing
	Error-value=TBD1: SPEAKER-ENTITY-ID TLV missing	This document
24	LSP instantiation error
	Error-value=TBD5: No PCEP session with the headend	This document

PCEP TLV Type Indicators IANA is requested to allocate new TLV Type Indicator values within the "PCEP TLV Type Indicators" registry, as follows:

Value	Meaning	Reference
TBD2	ORIGINAL-LSP-DB-VERSION TLV	This document
TBD3	PCEP-PATH-VECTOR TLV	This document

STATEFUL-PCE-CAPABILITY TLV IANA is requested to allocate a new bit value in the "STATEFUL-PCE-CAPABILITY TLV Flag Field" registry, as follows:

Bit	Description	Reference
TBD4	INTER-PCE-CAPABILITY	This document

Notification Object IANA is requested to allocate a new Notification Type and Notification Values within the "Notification Object" registry, as follows:

Notification-type	Meaning	Reference
TBD6	Inter-PCE State-sync	This document
	Notification-value=1: Add PCC's Open Information	This document
	Notification-value=2: Remove PCC's Open Information	This document

References Normative References Informative References

Examples The examples in this appendix are for illustrative purposes only, to show how the behavior of the state-sync inter-PCE session works.

Example 1 - Successful disjoint paths (requiring reroute)

Disjoint Paths Requiring Reroute PCC2 +----------+ / D=1 / +---------+ +---------+ | PCE1 |----| PCE2 | +---------+ +---------+ / D=1 / +----------+ | PCC3 | LSP : PCC3->PCC4 +----------+ PCE1 computation priority 100 PCE2 computation priority 200 ]]> Consider the PCEP sessions in , where computation priority is global for all the LSPs, and a link-disjoint path between LSPs PCC1->PCC2 and PCC3->PCC4 is required. Consider the PCC1->PCC2 is configured first, and PCC1 delegates the LSP to PCE1, but as PCE1 does not have the highest computation priority, it sub-delegates the LSP to PCE2 by sending a PCRpt with D=1 and including the SPEAKER-ENTITY-ID TLV over the state-sync session. PCE2 receives the PCRpt, and as it has delegation for this LSP, it computes the shortest path: R1->R3->R4->R2->PCC2. It then sends a PCUpd to PCE1 (including the SPEAKER-ENTITY-ID TLV) with the computed ERO. PCE1 forwards the PCUpd to PCC1 (removing the SPEAKER-ENTITY-ID TLV). PCC1 acknowledges the PCUpd by a PCRpt to PCE1. PCE1 forwards the PCRpt to PCE2. When PCC3->PCC4 is configured, PCC3 delegates the LSP to PCE2, PCE2 can compute a disjoint path as it has knowledge of both LSPs and has delegation also for both. The only solution found is to move PCC1->PCC2 LSP on another path, PCE2 can move PCC1->PCC2 as it has sub-delegation for it. It creates a new PCUpd with a new ERO: R1->R2-PCC2 toward PCE1 which forwards to PCC1. PCE2 sends a PCUpd to PCC3 with the path: R3->R4->PCC4. In this setup, PCEs are able to find a disjoint path, while without state-sync and computation priority, they could not.

Example 2 - Successful disjoint paths (simultaneous turnup)

Disjoint Paths with Simultaneous Turnup PCC2 +----------+ / \ D=1 / \ D=0 +---------+ +---------+ | PCE1 |----| PCE2 | +---------+ +---------+ D=0 \ / D=1 \ / +----------+ | PCC3 | LSP : PCC3->PCC4 +----------+ PCE1 computation priority 200 PCE2 computation priority 100 ]]> In this example (see ), suppose both LSPs are configured almost at the same time. PCE2 sub-delegates PCC3->PCC4 to PCE1 while PCE1 keeps delegation for PCC1->PCC2, PCE1 computes a path for PCC1->PCC2 and PCC3->PCC4, and can achieve disjointness computation easily. No computation loop happens in this case.

Example 3 - Unfeasible disjoint paths (insufficient state-sync sessions)

Unfeasible Disjoint Paths PCC2 +----------+ / D=1 / +---------+ +---------+ +---------+ | PCE1 |----| PCE2 |----| PCE3 | +---------+ +---------+ +---------+ / D=1 / +----------+ | PCC3 | LSP : PCC3->PCC4 +----------+ PCE1 computation priority 100 PCE2 computation priority 200 PCE3 computation priority 300 ]]> With the PCEP sessions as in , consider the need to have link disjoint LSPs PCC1->PCC2 and PCC3->PCC4. Suppose PCC1->PCC2 is configured first, PCC1 delegates the LSP to PCE1, but as PCE1 does not have the highest computation priority, it will sub-delegate the LSP to PCE2 (as it is not aware of PCE3 and has no way to reach it). PCE2 cannot compute a path for PCC1->PCC2 as it does not have the highest priority and is not allowed to sub-delegate the LSP again toward PCE3 as per . When PCC3->PCC4 is configured, PCC3 delegates the LSP to PCE2 that performs sub-delegation to PCE3. As PCE3 will have knowledge of only one LSP in the group, it cannot compute disjointness and can decide to fall back to a less constrained computation to provide a path for PCC3->PCC4. In this case, it will send a PCUpd to PCE2 that will be forwarded to PCC3. Disjointness cannot be achieved in this scenario because of lack of state-sync session between PCE1 and PCE3, but no computation loop happens. Thus it is required for all PCEs that support state-sync to have full mesh sessions between each other ().

Using Computation Priority and State-Sync Sessions to Increase Scaling The computation priority and state-sync session architecture can be used to increase the scaling of the PCE architecture (). Using computation priorities and state-sync sessions may allow to create groups of PCEs that manage a subset of the PCCs and perform some or no path computations. Decoupling PCEP session maintenance and computation will allow increasing scaling of the PCE architecture.

Improved Scalability In , two groups of PCEs are created: PCE1/2 maintain PCEP sessions with PCC1 up to PCC500, while PCE3/4 maintain PCEP sessions with PCC501 up to PCC1000. All four PCEs form one state-sync PCE set with a full mesh of state-sync sessions. A granular computation priority policy is set up as follows to load-share computation between PCEs:

PCE1 has priority 200 for association ID 1 up to 300, association source 0.0.0.0. All other PCEs have a decreasing priority for those associations.
PCE3 has priority 200 for association ID 301 up to 500, association source 0.0.0.0. All other PCEs have a decreasing priority for those associations.

If some PCCs delegate LSPs with association ID 1 up to 300 and association source 0.0.0.0, the receiving PCE (if not PCE1) will sub-delegate the LSPs to PCE1. PCE1 becomes responsible for the computation of these LSP associations while PCE3 is responsible for the computation of another set of associations. The procedures described in this document could help greatly in load-sharing between a group of stateful PCEs (see ).

Scenarios This appendix provides several scenarios for illustrative purposes, based on the topology of . There are many other cases where the solution defined in this document is also applicable.

Scenario 1 In the normal case (PCE1 as active PCE), consider that PCC1->PCC2 LSP is configured first with the link disjointness constraint, PCE1 sends a PCUpd message to PCC1 with the Explicit Routing Object (ERO): R1->R3->R4->R2->PCC2 (shortest path). PCC1 signals and installs the path. When PCC3->PCC4 is configured, the PCEs already knows the path of PCC1->PCC2 and can compute a link-disjoint path: the solution requires to move PCC1->PCC2 onto a new path to let room for the new LSP. PCE1 sends a PCUpd message to PCC1 with the new ERO: R1->R2->PCC2 and a PCUpd to PCC3 with the following ERO: R3->R4->PCC4. In the normal case, there is no issue for PCE1 to compute a link-disjoint path.

Scenario 2 Consider that PCC1 lost its PCEP session with PCE1 (all other PCEP sessions are UP) as shown in . PCC1 delegates its LSP to PCE2.

Scenario 2 PCC2 +----------+ \ \ D=1 +---------+ +---------+ | PCE1 | | PCE2 | +---------+ +---------+ D=1 \ / D=0 \ / +----------+ | PCC3 | LSP: PCC3->PCC4 +----------+ ]]> Consider that the PCC1->PCC2 LSP is configured first with the link disjointness constraint, PCE2 (which is the new active PCE for PCC1) sends a PCUpd message to PCC1 with the ERO: R1->R3->R4->R2->PCC2 (shortest path). When PCC3->PCC4 is configured, PCE1 is not aware of LSPs from PCC1 anymore, so it cannot compute a disjoint path for PCC3->PCC4 and will send a PCUpd message to PCC3 with the shortest path ERO: R3->R4->PCC4. When PCC3->PCC4 LSP will be reported to PCE2 by PCC3, PCE2 will ensure disjointness computation and will correctly move PCC1->PCC2 (as it owns delegation for this LSP) on the following path: R1->R2->PCC2. With this sequence of events and these PCEP sessions, disjointness is ensured.

Scenario 3

Scenario 3 PCC2 +----------+ / \ D=1 / \ D=0 +---------+ +---------+ | PCE1 | | PCE2 | +---------+ +---------+ / D=1 / +----------+ | PCC3 | LSP: PCC3->PCC4 +----------+ ]]> Consider the PCEP sessions in , and the PCC1->PCC2 LSP is configured first with the link disjointness constraint, PCE1 computes the shortest path as it is the only LSP in the disjoint association group that it is aware of: R1->R3->R4->R2->PCC2 (shortest path). When PCC3->PCC4 is configured, PCE2 must compute a disjoint path for this LSP. The only solution found is to move PCC1->PCC2 LSP on another path, but PCE2 cannot do it as it does not have delegation for this LSP. In this setup, PCEs are not able to find a disjoint path.

Scenario 4

Scenario 4 PCC2 +----------+ / \ D=1 / \ D=0 +---------+ +---------+ | PCE1 | | PCE2 | +---------+ +---------+ D=0 \ / D=1 \ / +----------+ | PCC3 | LSP: PCC3->PCC4 +----------+ ]]> Consider the PCEP sessions in , and that PCEs are configured to fall-back to the shortest path if disjointness cannot be found as described in . The PCC1->PCC2 LSP is configured first, PCE1 computes the shortest path as it is the only LSP in the disjoint association group that it is aware of: R1->R3->R4->R2->PCC2 (shortest path). When PCC3->PCC4 is configured, PCE2 must compute a disjoint path for this LSP. The only solution found is to move PCC1->PCC2 LSP on another path, but PCE2 cannot do it as it does not have delegation for this LSP. PCE2 then provides the shortest path for PCC3->PCC4: R3->R4->PCC4. When PCC3 receives the ERO, it reports it back to both PCEs. When PCE1 becomes aware of the PCC3->PCC4 path, it recomputes the constrained shortest path first (CSPF) algorithm and provides a new path for PCC1->PCC2: R1->R2->PCC2. The new path is reported back to all PCEs by PCC1. PCE2 also recomputes CSPF to take into account the new reported path. The new computation does not lead to any path update.

Scenario 5

Scenario 5 Now, consider a new network topology in with the same PCEP sessions as the previous example. Suppose that both LSPs are configured almost at the same time. PCE1 will compute a path for PCC1->PCC2 while PCE2 will compute a path for PCC3->PCC4. As each PCE is not aware of the path of the second LSP in the association group (not reported yet), each PCE is computing the shortest path for the LSP. PCE1 computes ERO: R1->PCC2 for PCC1->PCC2 and PCE2 computes ERO: R3->R1->PCC2->PCC4 for PCC3->PCC4. When these shortest paths will be reported to each PCE, each PCE will recompute disjointness. PCE1 will provide a new path for PCC1->PCC2 with ERO: PCC1->PCC2. PCE2 will also provide a new path for PCC3->PCC4 with ERO: R3->PCC4. When those new paths will be reported to both PCEs, this will trigger CSPF again. PCE1 will provide a new more optimal path for PCC1->PCC2 with ERO: R1->PCC2 and PCE2 will also provide a more optimal path for PCC3->PCC4 with ERO: R3->R1->PCC2->PCC4. So we come back to the initial state. When those paths are reported to both PCEs, this will trigger CSPF again. An infinite loop of CSPF computation is then happening with a permanent flap of paths because of the split-brain situation. Another common example to note would be two LSPs with link-diverse paths that share a common node in their path but are delegated to different PCEs. In case of the common node failure, both PCEs would detect the same, and each could independently compute a new path that might both choose the same new link. This permanent computation loop comes from the inconsistency between the state of the LSPs as seen by each PCE due to the split-brain: each PCE is trying to modify at the same time its delegated path based on the last received path information, which de facto invalidates this received path information.

Scenario 6

Scenario 6 In the example in , suppose that the disjoint LSPs from PCC1 to PCC2 and from PCC4 to PCC3 are created. All the PCEs have the knowledge of both domain topologies (e.g. using BGP-LS ). For operation/management reasons, each domain uses its own group of redundant PCEs. PCE1/PCE2 in domain 1 have PCEP sessions with PCC1 and PCC3 while PCE3/PCE4 in domain 2 have PCEP sessions with PCC2 and PCC4. As PCE1/2 does not know about LSPs from PCC2/4, and PCE3/4 do not know about LSPs from PCC1/3, there is no possibility to compute the disjointness constraint. This scenario can also be seen as a split-brain scenario. This multi-domain architecture (with multiple groups of PCEs) can also be used in a single domain, where an operator wants to limit the failure domain by creating multiple groups of PCEs, maintaining a subset of PCCs. As for the multi-domain example, there will be no possibility to compute the disjoint path starting from the head-ends managed by different PCE groups. In this document, we specify a solution that addresses the possibility to compute LSP association-based constraints (like disjointness) in split-brain scenarios while preventing computation loops.

Contributors