High speed data express in IP: Concept, Reference Architecture and Technologies

Abstract

With the rapid development of AI technology, intelligent computing and super-computing services, the demand for wide-area transmission of massive data is increasing. This draft makes full use of the advantages of IP network, such as statistical reuse and elastic supply, and proposes a reference architecture of high elasticity and high throughput data express based on IP network.¶

2. Reference Architecture

The data express solution enables massive data transmission from the data source to the destination. Data sources are devices or systems that generate or store large amounts of data, such as enterprise servers, databases, and storage devices. Data sources can be multiple nodes distributed across different geographies and network environments, or they can be a single node concentrated in one location. Data destination refers to the device or system that receives or processes a large amount of data, such as cloud computing platform, intelligent computing center, and supercomputing center. The data destination can be a public service provided by a third party or a cloud data center built by a data express operator.¶

Based on the massive transmission requirements from the data source to the destination end, the overall architecture of the data express solution is designed according to the hierarchical idea, which mainly includes three parts: cloud private network infrastructure, network controller and data express operation platform, as shown in¶

                         +--------------------+
                         |                    |
                         |    Data Express    |
          Task  -------> | Operation Platform |
        Delivery         |                    |
                         +---------+----------+
                                   |
                        +----------+-----------+
                        |                      |
        +---------------+  Network Controller  +---------+
        |               |                      |         |
        |               +-----------+----------+         |
        |                           |                    |
        |                           |                    |
+-------+---------------------------+--------------------+----------+
|                                                                   |
| +-----------+                                       +-----------+ |
| |Data Source|                                       |  Super    | |
| |  Memory   |                                +------+ Computing | |
| | card/disk |                Wide Area       |      |  Center   | |
| +-----+-----+     +------+Interconnection +--+--+   +-----------+ |
|       |           |Cloud +----------------+Cloud|                 |
|       |           |  PE  |                | PE  |                 |
|       |           +---+--+                +--+--+   +-----------+ |
| +-----v------+        |                      |      |  General  | |
| |Data Express|        |                      +------+ Computing | |
| |  Station   +--------+                             |   Center  | |
| +------------+                                      +-----------+ |
|                                                                   |
|                      Infrastructure                               |
+-------------------------------------------------------------------+

• Infrastructure¶

  • Data Express can use the existing network infrastructure to provide data transmission services for customers, and can access different data source and data consumption end through SPN, PON, optical fiber direct connection and other means according to the situation of live network resources, so as to achieve dynamic scheduling and optimization of network resources and ensure the quality and efficiency of data transmission. SPN is the recommended access mode for data express because of its large access bandwidth, elastic bandwidth configuration, and wide coverage.¶

• Network Controller¶

• Network controllers include SPN and other network controllers in different fields, which complete the network configuration according to the requirements of the service platform, and realize the whole process automation and intelligent opening of the data express service. The main functions include:¶

  • Network configuration: Automatically or manually configure parameters such as network topology, routes, and security policies based on network requirements and scenarios to achieve flexibility and customization.¶

  • Information collection: Collects and analyzes network performance, quality, and security indicators in real time to optimize and improve data express services.¶

  • Network monitoring: Monitors the network in real time to detect and handle anomalies such as network faults, attacks, and leaks, ensuring data transmission security and stability.¶

• Operation Platform¶

  • Operation platform refers to the online platform that provides users with one-stop data express service management and monitoring functions, and provides users with business ordering, opening, monitoring, billing and other functions. The operation platform can be accessed through the web or mobile terminal to realize the visualization and intelligence of the whole process of data express business. The main functions include:¶

    • Service order: Users can order data express services online on the operation platform, and select parameters such as data source, destination, transmission mode, and transmission time to generate service orders.¶

    • Enabling the service: Users can enable the online data express service on the operation platform. Based on the order information, users can automatically or manually configure data sources, destinations, and network resources to start data transfer tasks.¶

    • Service monitoring: Users can monitor data express services online on the operation platform and view data transfer progress, status, rate, and quality in real time to detect and handle anomalies in a timely manner.¶

    • Service charging: Users can charge online data express services on the operating platform, and generate and pay bills based on preset charging rules based on data transfer volume, duration, and bandwidth.¶

3. Core Operation

This section elaborates the workflow and interaction logic of functional components defined in the reference architecture of Section 2. The whole operation follows the high-speed IP data express(HSDE) objective introduced in Section 1, realizing task scheduling, rate regulation and RDMA transmission support within the constraints of the reference architecture.¶

An HSDE task represents a discrete transmission request submitted by an upper-layer application. Each task is associated with a set of administrative parameters including priority, target SLA, resource quota, and destination endpoint identifier.¶

The HSDE Task Manager performs admission control for incoming tasks. A task MAY be admitted only if sufficient quota and QP resources are available. Once admitted, the task enters the ready queue waiting for transmission arbitration by the scheduler. Tasks that fail admission MAY be queued for later retry or rejected according to implementation policy.¶

The HSDE scheduler is the central arbitration component. It takes candidate rate recommendations from congestion control, applies per-task SLA constraints, remaining resource quotas, and per-QP rate guard bounds, and computes a final target sending rate for each selected task.¶

The scheduler serializes transmission opportunities across competing HSDE tasks. Rate outputs from the scheduler are delivered to the RDMA sender for packet posting. The scheduler does not interpret the source of transport feedback; it operates on abstract loss, RTT, and progress signals.¶

The congestion control module computes a desired sending rate based on feedback samples collected from the transport datapath. This rate recommendation is submitted to the HSDE scheduler for final arbitration. The congestion control module MUST respect the rate guard limits associated with the target QP. The scheduler retains the authority to cap or adjust the recommended rate to satisfy SLA and quota rules.¶

HSDE maintains a shared pool of reusable RC QPs to serve concurrent tasks. The QP pool implements the transport resource management capability described in the reference architecture of Section 2. A QP is assigned to one HSDE task at a time. When a task completes or is preempted, the QP context is reset and returned to the pool for reassignment to a subsequent task.¶

QP reset is implementation-specific, but it MUST clear all task-specific transport state associated with the prior HSDE task before the QP is reused.¶

The baseline feedback mechanism for HSDE uses native RDMA RC AETH acknowledgments, corresponding to the RDMA feedback path in the Section 2 architecture. The receiver generates standard AETH completion messages to report acknowledged PSN progress. These AETH messages are processed by the sender to advance send credits, update RTT estimates, and detect packet loss.¶

Loss signals and acknowledgment progress from baseline AETH are delivered into the HSDE Feedback Collector for consumption by congestion control and scheduler logic.¶

4. Windowless ACK Extension

As illustrated in the reference architecture of Section 2, HSDE defines a standardized interface for RDMA feedback. The T/C-AETH specified herein is an optional alternative implementation of this feedback component, targeting the massive high-speed data transmission goal introduced in Section 1. This extension preserves the layering, component partition and inter-component interfaces of the original reference architecture.¶

HSDE targets high-throughput RDMA transmission over IP networks. Native RC AETH acknowledgments introduce non-negligible control-plane overhead under high line rates. The traditional ACK windowing logic can also increase feedback latency, which degrades the responsiveness of congestion control for large-flow HSDE tasks.¶

The T/C-AETH mechanism is designed to reduce acknowledgment overhead and deliver timely loss and timing feedback for HSDE flows, while remaining compatible with the core HSDE scheduling and rate-control architecture.¶

Support for T/C-AETH is a per-QP capability negotiated during QP setup. Negotiation uses private or vendor-specific QP attributes. No new standard RDMA attributes are defined by this document.¶

If both initiator and responder signal support for T/C-AETH, the QP MAY enter windowless ACK mode. If negotiation fails, the endpoints MAY retry negotiation within the same QP-setup procedure. Once T/C-AETH mode is successfully enabled for a QP, the ACK mode MUST NOT change for the remaining lifetime of that QP. A QP that does not enable T/C-AETH uses the baseline RC AETH feedback path.¶

This specification defines only the logical fields of a T/C-AETH message. It does not mandate a fixed binary wire encoding or bit-level layout. Implementations MAY select suitable encoding schemes.¶

A valid T/C-AETH logically contains:¶

• Highest acknowledged PSN: the sequence number representing the right edge of contiguous acknowledged packets.¶

• RTT timestamp: a time sample for round-trip measurement.¶

• Compact segment list: a list of PSN intervals identifying detected packet loss at the receiver. The list MAY be empty.¶

An empty segment list indicates that the receiver reports acknowledgment progress and timing information, with no newly detected loss intervals.¶

When operating in T/C-AETH mode, the receiver maintains an ongoing list of detected loss intervals for the QP. When new acknowledgment progress covers existing loss intervals, the covered intervals are removed from the pending segment list.¶

If the local storage allocated for the segment list is exhausted, older loss intervals MUST be discarded first; newer loss intervals are retained with higher priority. Any loss information discarded this way relies on the standard RDMA timeout mechanism for recovery.¶

T/C-AETH SHOULD be piggybacked onto outgoing RDMA response packets when a response is available. If no response packet is pending, the receiver MAY send a standalone T/C-AETH control packet. Standalone T/C-AETH packets SHOULD be assigned higher transmit priority to reduce feedback latency.¶

Malformed or out-of-spec T/C-AETH packets received by the receiver MUST be silently discarded. Discarding such packets MUST NOT alter QP state. Recovery for any associated loss depends on RDMA timeout.¶

Upon receiving a valid T/C-AETH message, the sender updates its tracked highest acknowledged PSN by taking the maximum value among all received highest-ack-psn fields.¶

The sender translates the segment list, acknowledged progress, and timestamp into loss events, progress updates, and RTT samples. These signals are forwarded to the HSDE Feedback Collector.¶

When a QP operating in T/C-AETH mode is reused for a new HSDE task, all T/C-AETH-specific context associated with the prior task MUST be cleared before the new task begins transmission.¶

Feedback produced by T/C-AETH is injected into the same HSDE Feedback Collector defined in the Section 2 architecture. The scheduler and congestion control module consume abstract feedback signals and remain agnostic to whether the underlying acknowledgment comes from standard AETH or from T/C-AETH.¶

The T/C-AETH extension does not modify the HSDE scheduler arbitration rules, quota enforcement logic, or QP pool lifecycle semantics described above.¶

Lost T/C-AETH control messages do not trigger a dedicated recovery procedure. Unresolved loss detection and recovery for the flow MUST rely on the native RDMA timeout mechanism defined for RC QPs.¶

5. Summary

TBD¶

8. Normative References

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>.