BIER in AI considerations

Introduction Large Language Models (LLMs) are currently a hot topic in artificial intelligence. The training and inference processes of LLMs typically consume significant GPU resources. During this process, there is a large amount of data flow between GPUs. This data flow is based on basic AI operations such as Broadcast, Scatter, Gather, and Reduce, as well as further variations of these basic communication primitives such as AllReduce, AllGather, ReduceScatter, and AlltoAll. Some of these operations, such as Broadcast and AllGather, involve sending the same data to multiple different nodes. This one-to-many sending is very similar to multicast operations in network layers. With the development of large-scale models, the MoE architecture is a widely used sparse model architecture. It allows different GPUs to handle different model parameters instead of the entire model. By introducing a multi-GPU collaborative computation model, it achieves an effect similar to the original dense model, but with significantly reduced GPU load, making it a highly cost-effective choice (see the ). The MoE architecture can be used for both training and inference. When processing tokens, it selects multiple experts to send them, which is very similar to multicast. In addition, discussions are underway regarding the implementation of INC (In-Network Collective/Computing) based on BIER (see the ); in particular, hierarchical INC may find application in networks operating at the scale of millions of GPUs. These applications utilize "ingress-defined replication" initiated at the ingress or source side. This means that the application layer at the ingress/source side has the autonomy to determine the set of destinations to which traffic is sent, without the need for maintaining flow-state. In current applications, these one-to-many sending are actually transmitted via unicast at the network layer, which can easily lead to the same data consuming bandwidth through the same link. However, network layer multicast technologies, including BIER and PIM, are already valuable for deployment and application. Therefore, by leveraging network layer multicast technology, bandwidth can be saved in one-to-many data flow scenarios in AI Fabric. One-to-many data flows in AI tasks may involve different scenarios, and different AI tasks will be distributed across different GPU sets as needed. Some tasks may require only a small number of GPUs, while others require a larger number. As described in , some AI tasks involve a large number of GPUs in their one-to-many data flows; for example, the token dispatch expert pool under the LLM MoE architecture may reach 256 or more. Furthermore, the selection of destination experts is highly dynamic, meaning that the destination expert set is different for each token. BIER technology, due to its stateless nature, is the only network layer multicast technology suitable for this high dynamic characteristic. BIER can be used as the network layer transport technology for all one-to-many data flows in AI tasks. This draft summarizes and analyzes the considerations for applying multicast technology to AI computing scenarios, providing a reference for the implementation in this direction.

Considerations

Coordinated Modification with Application Layer Interface Calls To achieve ingress-defined replication using network-layer multicast, the application layer needs to modify its interface code, changing from unicast calls to multicast calls. The original point-to-point management method is no longer suitable and needs to be adapted to a multicast-oriented approach. This modification may render existing optimization techniques for point-to-point unicast inapplicable. For BIER technology, its multicast encapsulation method is simple: clearly define the set of destinations to be sent and encapsulate their corresponding IDs in the BIER header. Each bit encapsulated in the BitString of the BIER header corresponds to an actual destination GPU. When the application layer can directly encapsulate the BIER header, it can either call the network layer encapsulation module to encapsulate the BIER header directly, or inform modules such as the NIC so that these modules can encapsulate the BIER header. also provides a method that allows BIER header encapsulation to be performed on the switch as needed.

Changes in RDMA Operations Brought About by Multicast RDMA operations are commonly used in AI computing, with one-sided operations such as Write and Read significantly improving information retrieval efficiency. Applying multicast technology to this scenario brings about a completely different change from traditional point-to-point management. In particular, while Write operations in unicast write to inconsistent addresses on different destination GPUs, in multicast scenarios, writing to the same contiguous address is ideal for more efficient processing. Current implementations may employ various methods to optimize multicast writes. For example, the physical addresses written by different destination GPUs may differ, but the virtual addresses they represent are the same. This avoids the problem of converting between multiple physical addresses when the source GPU initiates a write operation. When all destination GPUs can be allocated the same physical address for writing, write efficiency can be further improved. When applying multicast to two-sided operations, this problem may not arise.

ECMP and Adaptive Routing DCs use massive ECMP and tree-based multicast inherently cannot take advantage of that dynamically in the data plane. Each tree is nailed by signaling to a single path, even though different trees can be signaled along different paths. The importance of the data plane ECMP is clearer if we use the unicast comparison: when there are multiple paths to reach two different prefixes, it is desired for both prefixes to use all the paths, instead of pinning prefix 1 to path 1 and prefix 2 to path 2. BIER does not have this problem. There are no explicit trees and forwarding follows unicast paths, and ECMP can be inherently used in the data plane. At each hop, the lookup for a BFER results in a set of downstream BFR neighbors, and any of them can be used. Adaptive routing is another thing that tree-based multicast cannot do, yet easily achieved with BIER, as described in .

Multicast Reliability Analysis AI computing is highly sensitive to packet loss, not only actual packet loss but also delays caused by congestion. Regardless of the multicast technology used, reliability must be addressed. Unicast technologies like DCQCN are difficult to directly apply to multicast scenarios in AI computing. If the source side slows down the transmission simply because one receiver missed a message, multicast transmission efficiency will be significantly reduced. As learned from Chapter 3.13, "Reliable Multicast," of the IETF has conducted research on reliable multicast at the network layer, but this cannot be directly applied to AI Fabric. When applying multicast technology to AI Fabric, several considerations exist. For example, in addition to multicast distribution, the reception status of the next hop can be aggregated, and this operation can be performed hop-by-hop, with feedback sent back to the source. When a packet is intentionally not received, determining packet loss and providing hop-by-hop feedback can significantly impact efficiency. Therefore, congestion-based or link failure-based detection may be more helpful in early detection of packet loss, with prevention being more effective than post-event remediation. Therefore, when applying multicast technology, the efficiency of the application layer sending unicast packets for acknowledgment after receiving a packet is higher than that of the network layer. When congestion or link failure occurs during packet forwarding, the Trim function can be used to provide feedback. For BIER, the bitmap carried in the header of the packet sent to the next hop indicates the affected destination. Therefore, sending the Trim packet containing the bitmap to the source GPU can make the source retransmit the packet only for the affected destination, shortening the recovery time. BIER OAM techniques, such as Ping () and PMMM (), can be used to monitor the quality of BIER transmission paths within AIDC; they can provide Out-of-band function. Furthermore, CSIG () provides in-band functionality and can be integrated seamlessly with BIER. As a Layer 2.5 technology, BIER header can follow Layer 2 CSIG tag and be transmitted alongside the data payload. The return-path L4+ CSIG message can encapsulate a BIER packet or directly use the L4+ packet encapsulation method. By initiating periodic checks from the application layer, it is entirely feasible to get the quality of BIER forwarding paths in AIDC and, in the event of congestion, make timely adjustments to parameters like Entropy that influence these paths, thereby ensuring the smooth flow of BIER traffic. When combined with Trim technology, this approach covers all failure scenarios, consistently guaranteeing the reliability of BIER transmission.

BIER Forwarding and Encapsulation The bit-indexed forwarding mechanism defined in is applicable in any case where the destination is represented by bits in a BitString or Bitmap. This is the core mechanism of BIER technology. In typical scenarios, standard BIER bitstring encapsulation is fully sufficient to meet requirements; specifically, a standard 256-bit bitstring implementation can satisfy the encapsulation needs for AI computing across the vast majority of scale-up and scale-out scenarios. In scenarios where the pool of potential destination is significantly larger than 256, yet the selected targets are extremely sparse, using the 256-bitstring encapsulation method might require multiple BIER header encapsulations distinguished by SI. proposes a sparse encapsulation method that can encapsulate sparse destinations as a list in the BitString positions. In some implementations that are further optimized to meet specific needs, some fields in the BIER header defined in might be omitted. For example, in some AI computation, there are many small packet (tens of bytes) interaction scenarios. Unused fields in the BIER header can be removed to reduce byte usage and improve forwarding efficiency. In some existing implementations, in-network computing and other operations may be combined with BitString, where BitString is still used to represent the destination set.

Considerations for Future Scalability To meet the demands of larger-scale AI computing, the number of GPUs employed may increase further, potentially reaching millions within a single data center. Proposals have also emerged to extend AI training and inference tasks across multiple data centers located in different geographical areas. With BIER, the scaling is about the number of BFERs in the domain. The BIER Forwarding Tables (BIFTs) hold one entry for each BFER, though the entries are organized into different BIFTs. From the amount of state point of view, 65k total entries (required for the 16-bit per-subdomain BFR-ID) across all the tables are still a small number compared to unicast routes. Of course, the BitStringLength is limited and it is typically 256. To send to more than BitStringLength BFERs, multiple copies are needed, one for each set of BitStringLength BFERs. In this document, this is referred to as the "scale-out" option (with multiple copies). Another option is dividing the entire network into per-region sub-domains to use tunnel segmentation (Section 4, . A segmentation point belongs to multiple sub-domains, decapsulating the BIER packets received in an upstream region and re-encapsulating for downstream regions. In this document, it is also referred to as "scale-across" option (across regions) which is used to enable cross-DC federated computing.