Network Working Group Z. Li Internet-Draft Z. Du Intended status: Standards Track China Mobile Expires: 6 May 2027 J. Wang W. Cheng Y. Zhao Centec C. Zhao SAIA X. Sun Inesa 2 November 2026 Key-Value (KV) Cache Block Transfer Protocol (KBTP) over RDMA: Block Load, Mapped Store, and Unmapped Store draft-li-rdma-kvcache-transfer-optimization-00 Abstract Inference systems move key-value (KV) cache blocks between instances and cache tiers over RDMA. Existing practice forces a choice between statically registered remote memory, which is fast but wastes memory, and per-transfer remote procedure calls, which use memory well but involve the target's CPU in every transfer. This document specifies the KV Cache Block Transfer Protocol (KBTP), an upper-layer protocol over RDMA with three operations: Block Load, which places a stored block into the initiator's memory; Block Mapped Store, which writes a block into an already allocated block; and Block Unmapped Store, which allocates a free block at the target and writes it in one operation. KBTP defines the request and completion messages, a block table, and a free-block ring whose counters use masked atomic fetch- and-add. These structures are designed so that a target can execute loads and stores within its RDMA network interface, and allocations as well where the interface supports conditional execution, without involving its CPU per operation, while remaining interoperable with targets that process requests in software. KBTP is bound to RDMAP with the RDMA protocol extensions, and can also be used over InfiniBand and RoCEv2. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Li, et al. Expires 6 May 2027 [Page 1] Internet-Draft KV Cache Block Transfer over RDMA November 2026 Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 6 May 2027. Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Background . . . . . . . . . . . . . . . . . . . . . . . 3 1.2. Problem Statement . . . . . . . . . . . . . . . . . . . . 3 1.3. Overview . . . . . . . . . . . . . . . . . . . . . . . . 4 1.4. Scope and Relationship to Other Work . . . . . . . . . . 5 2. Conventions and Definitions . . . . . . . . . . . . . . . . . 5 3. Target Memory Structures . . . . . . . . . . . . . . . . . . 6 3.1. Block Pool . . . . . . . . . . . . . . . . . . . . . . . 6 3.2. Block Table . . . . . . . . . . . . . . . . . . . . . . . 6 3.3. Free-Block Ring . . . . . . . . . . . . . . . . . . . . . 7 3.4. Allocation Quotas . . . . . . . . . . . . . . . . . . . . 8 4. Session Establishment . . . . . . . . . . . . . . . . . . . . 9 5. Messages . . . . . . . . . . . . . . . . . . . . . . . . . . 11 5.1. Common Header . . . . . . . . . . . . . . . . . . . . . . 11 5.2. Descriptors . . . . . . . . . . . . . . . . . . . . . . . 12 6. Operations . . . . . . . . . . . . . . . . . . . . . . . . . 13 6.1. Block Load . . . . . . . . . . . . . . . . . . . . . . . 13 6.2. Block Mapped Store . . . . . . . . . . . . . . . . . . . 13 6.3. Block Unmapped Store . . . . . . . . . . . . . . . . . . 13 6.4. Block Free . . . . . . . . . . . . . . . . . . . . . . . 14 Li, et al. Expires 6 May 2027 [Page 2] Internet-Draft KV Cache Block Transfer over RDMA November 2026 6.5. Completion . . . . . . . . . . . . . . . . . . . . . . . 14 6.6. Direct Modes . . . . . . . . . . . . . . . . . . . . . . 14 7. Status Codes . . . . . . . . . . . . . . . . . . . . . . . . 15 8. Target Execution Model . . . . . . . . . . . . . . . . . . . 16 9. Transport Bindings . . . . . . . . . . . . . . . . . . . . . 16 9.1. RDMAP Binding . . . . . . . . . . . . . . . . . . . . . . 16 9.2. InfiniBand and RoCEv2 . . . . . . . . . . . . . . . . . . 17 10. Operational Considerations . . . . . . . . . . . . . . . . . 17 11. Security Considerations . . . . . . . . . . . . . . . . . . . 18 12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 18 13.1. Normative References . . . . . . . . . . . . . . . . . . 18 13.2. Informative References . . . . . . . . . . . . . . . . . 19 Appendix A. Examples . . . . . . . . . . . . . . . . . . . . . . 21 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 21 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 21 1. Introduction 1.1. Background The KV cache that attention layers keep for processed tokens is reused across decoding steps and, increasingly, across requests. Longer context windows make it larger than accelerator memory can hold, so serving systems store it in several tiers and move it between prefill instances, decode instances, and cache servers [PAGEDATTENTION] [MOONCAKE]. Two organizations are common: a central KV store that inference instances write to and read from, and peer- to-peer transfer in which, for example, a prefill instance writes directly to the decode instance that will use the cache. In both, the data moves in fixed-size blocks (typically holding the cache of 16 to 512 tokens, from kilobytes to megabytes) over RDMA. 1.2. Problem Statement Three RDMA usage patterns are common today: 1. One-sided access to statically allocated memory: the target registers a large region in advance (often gigabytes) and the source writes or reads it directly. Access is fast, but the coarse static allocation leaves much memory unused. 2. One-sided access to dynamically allocated memory: the source asks the target, with a two-sided remote procedure call, to allocate a fine-grained region (for example 4 KiB to 1 MiB), then accesses it one-sidedly. Memory is used well, but every allocation involves the target CPU, which limits throughput and adds latency. Li, et al. Expires 6 May 2027 [Page 3] Internet-Draft KV Cache Block Transfer over RDMA November 2026 3. Two-sided transfer: the source sends the data in messages and the target CPU allocates memory and copies or registers it. Memory is used well, but every transfer involves the target CPU. +==================+===============+===============+===========+ | Approach | Memory | Target CPU | Operation | | | utilization | per operation | cost | +==================+===============+===============+===========+ | One-sided, | low | no | low | | static | | | | +------------------+---------------+---------------+-----------+ | One-sided, | medium | yes | medium | | dynamic | | (allocation) | | +------------------+---------------+---------------+-----------+ | Two-sided | high | yes | high | +------------------+---------------+---------------+-----------+ | Target-executed | high (fine- | not required | one round | | block operations | grained | per operation | trip per | | (this document) | blocks) | | operation | +------------------+---------------+---------------+-----------+ Table 1: Comparison of Approaches No interoperable protocol provides fine-grained allocation and one- round-trip block access without per-operation involvement of the target CPU. Since inference instances and cache servers are built by different implementers, the message formats and the structure of target memory must be specified, not only the RDMA operations. 1.3. Overview A KBTP target manages a pool of equally sized blocks. It keeps a block table that maps block handles to blocks, and a free-block ring that lists unallocated blocks. An initiator sends one request message per operation; the target answers with a completion. The operations are: Block Load: the target looks up the handle and writes the block into a buffer named by the initiator. Block Mapped Store: the target looks up the handle and writes data supplied by the initiator into the block. Block Unmapped Store: the target takes a free block from the ring, binds it to the handle, and writes the data into it. Block Free: the target unbinds the handle and returns the block to the ring. Li, et al. Expires 6 May 2027 [Page 4] Internet-Draft KV Cache Block Transfer over RDMA November 2026 Each operation needs a fixed, short sequence of RDMA operations at the target whose addresses come from the request, the table, or the ring. The formats in this document are designed so that the target can pre-post these sequences and let its RDMA interface execute them when a request arrives (Section 8). A target that processes requests in software is equally conformant; the initiator cannot tell the difference. Optional direct modes let initiators that the target trusts read blocks and allocate them with one-sided operations only (Section 6.6). 1.4. Scope and Relationship to Other Work KBTP defines an upper-layer protocol in the sense of RDMAP [RFC5040], like the RDMA transport for RPC [RFC8166] and iSER [RFC7145]. It is bound normatively to RDMAP over DDP [RFC5041] and MPA [RFC5044] [RFC6581], with the atomic and immediate data operations of [RFC7306]. A mapping to InfiniBand reliable connections and RoCEv2 [IBTA] is given in Section 9.2. KBTP does not define new RDMA operations. [I-D.talpey-rdma-commit] specifies RDMA extensions for remote persistence and visibility, which KBTP targets can use for persistent tiers. Block keys and cache indexing, which tell an initiator which handle holds a given block, are provided by [I-D.li-kvcache-efficient-storage] or by local means. Typical uses are: storing the KV cache of external knowledge (retrieval-augmented generation documents) once and loading it into any instance; keeping the cache of very long contexts in a shared pool and reusing it across requests; and caches embedded in network devices of computing power networks. 2. Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. This document uses the RDMAP terms Steering Tag (STag), Tagged Offset, Send, RDMA Write, RDMA Read, and Remote Peer from [RFC5040], and FetchAdd from [RFC7306]. In addition: Initiator: The endpoint that issues KBTP requests. Target: The endpoint that owns the block pool. Li, et al. Expires 6 May 2027 [Page 5] Internet-Draft KV Cache Block Transfer over RDMA November 2026 Block: A fixed-size region of the target's block pool, identified by its Block Offset, the Tagged Offset of its first octet within the Pool STag. Block Handle: A 64-bit name, chosen by an initiator within the handle range assigned to it, that is bound to a block by an Unmapped Store. Generation: A 16-bit counter of a table entry that changes whenever the handle is bound or unbound, so that stale references are detected. Sealed block: A block whose contents will not change until it is freed. All fields in messages and in memory structures read by remote peers are in network byte order, except the address fields listed in Section 4 when the target selects little-endian address fields. 3. Target Memory Structures 3.1. Block Pool The target allocates F blocks of Block Size octets each within a memory region named by the Pool STag. Each block is identified by its Block Offset. Tables and ring slots store Block Offsets rather than block numbers, so that a value read from them can be used directly as the address of an RDMA operation without arithmetic. 3.2. Block Table The block table has one 16-octet entry per handle that the target serves. The entry for handle h is at Table Offset + 16 * (h - Table Base Handle) within the Table STag. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Flags | Generation | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Block Offset (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 1: Block Table Entry Li, et al. Expires 6 May 2027 [Page 6] Internet-Draft KV Cache Block Transfer over RDMA November 2026 Flags: 0x0001 VALID (the handle is bound), 0x0002 SEALED. Data Length is the number of valid octets in the block. Generation never takes the value zero. An entry is written only by the target; when it binds an entry, the target writes Data Length and Block Offset before Flags and Generation. 3.3. Free-Block Ring The free-block ring has N slots, where N = 2^n is a power of two not smaller than F, and two 64-bit counters, Head (used to allocate) and Tail (used to free), each aligned to 8 octets. Each slot is 16 octets: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sequence (Ticket) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Block Offset (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 2: Ring Slot A slot whose Block Offset is all ones is empty (consumed). Each counter holds two fields that advance together: a 32-bit Ticket in bits 63-32 and the Slot Offset, the byte offset of the current slot (16 times the slot number), in the low n+4 bits. Bits 31 to n+4 are zero. Counters are modified only with FetchAdd Section 5.1.1 of [RFC7306] using: Add Data = 0x0000000100000010 Add Mask = bit 63 | bit 31 | bit (n+3) The Add Mask makes each field add independently and discard its carry, so one FetchAdd returns the current Ticket and Slot Offset and advances the Ticket by one and the Slot Offset by one slot, wrapping at the end of the ring. The returned Slot Offset can be used as an address offset without further arithmetic, which allows targets to execute the allocation in their RDMA interface. Counters are in the target's memory byte order, as RFC 7306 specifies for atomic operations; they are never read with RDMA Read. The rules are: Li, et al. Expires 6 May 2027 [Page 7] Internet-Draft KV Cache Block Transfer over RDMA November 2026 * Initialization: for s < F the target writes slot s = {Sequence s, Block Offset of the s-th free block}; for F <= s < N it writes {Sequence s - N (modulo 2^32), all ones}. It sets Head to Ticket 0 and Slot Offset 0, and Tail to Ticket F and Slot Offset 16 * (F mod N). * Allocation: FetchAdd on Head returns ticket t and offset o. The slot at o holds the block for this allocation once its Sequence equals t and its Block Offset is not all ones; until then the allocator re-reads the slot. Having taken the Block Offset, the slot is marked consumed by setting its Block Offset to all ones: by the target itself for target-executed allocations, and by the target when it processes BIND for direct allocations (Section 6.6). * Free: the target first updates the table entry (Section 6.4), then performs FetchAdd on Tail, obtaining ticket u and offset o. If the slot at o is consumed, the target writes the Block Offset and then Sequence u. If it is not yet consumed, because the allocation that owns it has taken its ticket but not yet read it, the target MUST defer the write until the slot is consumed; it MUST NOT overwrite the slot. With these rules no block is ever lost or handed out twice: each slot is written only after its previous content has been consumed, and each allocation reads exactly the slot written for its ticket. Writing the Block Offset before the Sequence ensures that a reader that observes the new Sequence also observes the new Block Offset. 3.4. Allocation Quotas The target assigns each initiator an Allocation Quota at session setup and keeps the sum of quotas not larger than F. An initiator's usage is the number of handles it has bound plus the number of tickets it has taken and not yet bound. An initiator MUST NOT exceed its quota; the target releases one unit of usage when a FREE completes or when an abandoned ticket is reclaimed. When all initiators respect their quotas, the ring always holds a block for every ticket taken, so an allocation waits at most for an in-progress free to finish writing its slot. Target-executed allocations take blocks from a ring shared by all initiators that the target serves this way. For DIRECT_ALLOC (Section 6.6), the target instead gives each initiator its own ring and Head counter over a partition of the pool whose size equals the initiator's quota, so that every ticket on that ring belongs to that initiator and the quota is enforced by the ring itself. A ticket that is not bound by BIND within the Ticket Timeout advertised in Li, et al. Expires 6 May 2027 [Page 8] Internet-Draft KV Cache Block Transfer over RDMA November 2026 HELLO_ACK is abandoned. Before reclaiming its block, the target MUST ensure that no RDMA Write of the initiator can still reach the block, by invalidating the initiator's Partition STag (Section 4), which disables DIRECT_ALLOC for the rest of the session, or by terminating the connection. It then consumes the slot, returns the block to the ring with a new ticket, and rejects a later BIND for that ticket with status TICKET_ABANDONED. If a target observes an allocation by an initiator that exceeds its quota, it completes the request with NO_SPACE and MAY terminate the session. 4. Session Establishment After the RDMA connection is established, the initiator sends HELLO and the target answers with HELLO_ACK, both as Send messages and both starting with the common header (Section 5.1) in which Request ID, Block Handle, Generation, and Status are zero (Status reports an error in HELLO_ACK). 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Requested Features | Completion Credits | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Initiator ID (128 bits) | | | | | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 3: HELLO Body Completion Credits is the number of receive buffers the initiator has posted for COMPLETION messages. The Initiator ID is a stable random identifier of the initiator; a target uses it to restore the initiator's handle range and bindings after a reconnection. Because it gives access to those bindings, the Initiator ID MUST be generated with at least 128 bits of entropy and kept secret, MUST be sent only on connections authenticated as described in Section 11, and a target MUST reject a HELLO whose Initiator ID belongs to a session that is still active. Li, et al. Expires 6 May 2027 [Page 9] Internet-Draft KV Cache Block Transfer over RDMA November 2026 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Features | Request Credits | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Block Size | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Max Inline | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Allocation Quota | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Handle Base (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Handle Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ticket Timeout (ms) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Direct-mode descriptors (present per Features, 16 each) ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 4: HELLO_ACK Body (Target Descriptor) Features: 0x0001 DIRECT_LOAD, 0x0002 DIRECT_ALLOC, 0x0004 RECORD_COMPLETION, 0x0008 ADDR_LE. Each direct-mode descriptor is 16 octets: STag (32 bits), Tagged Offset (64 bits), and Count (32 bits). If DIRECT_LOAD is set, a Table descriptor (Count = number of entries) follows, then a Table Base Handle (64 bits) and 8 reserved octets, and then a Pool descriptor (Count = F) that grants read access only. If DIRECT_ALLOC is set, a Partition descriptor follows, naming the part of the pool reserved for this initiator (Count = its quota) through an STag that grants write access, is specific to this connection (for example a memory window bound to it), and covers nothing else; then a Ring descriptor (Count = N) and a Head descriptor (Count = 1; the Tagged Offset MUST be a multiple of 8, as required for FetchAdd) of the ring dedicated to this initiator (Section 3.4). Handles of different initiators never overlap, so each initiator can choose handles for Unmapped Store without coordination. Any initiator MAY load any handle the target authorizes it to read; handles are exchanged among initiators by means outside this protocol. If ADDR_LE is set, the initiator MUST encode the Block Handle and the STag, Tagged Offset, and Length fields of the Data and Completion Descriptors in little-endian byte order, and the Block Offset fields of table entries and ring slots are in little-endian byte order. A Li, et al. Expires 6 May 2027 [Page 10] Internet-Draft KV Cache Block Transfer over RDMA November 2026 target sets ADDR_LE when its RDMA interface uses little-endian work request fields and it wants to place these fields into pre-posted work requests unchanged (Section 8). GOODBYE ends the session. With flag 0x0001 RELEASE, the target frees all handles bound by the initiator. Otherwise, and when a connection fails, the target keeps the initiator's bindings for a retention time of its choosing; an initiator that reconnects with the same Initiator ID within that time receives the same handle range and finds its bindings intact. 5. Messages 5.1. Common Header 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Version | Opcode | Flags | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Request ID (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Block Handle (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Generation | Status | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 5: KBTP Common Header (24 Octets) Version is 1. Opcodes are listed in Table 2. Request ID is chosen by the initiator and echoed in the completion. Generation in a request is the generation the initiator expects (zero means "any"); in a completion it is the entry's current generation. Status is zero in requests. Flags are interpreted per opcode; for LOAD, STORE, ALLOC_STORE, and FREE they are: * 0x0001 INLINE: data follows the request. * 0x0002 SEAL: seal the block after the store. * 0x0004 RECORD: complete with a completion record (Section 6.5); a Completion Descriptor is present. * 0x0008 INVALIDATE: the target completes with Send with Invalidate [RFC5040], invalidating the STag of the initiator's Data Descriptor. Li, et al. Expires 6 May 2027 [Page 11] Internet-Draft KV Cache Block Transfer over RDMA November 2026 * 0x0010 ALLOW_UNSEALED: a load may return an unsealed block. +========+=============+===========================================+ | Opcode | Name | Body after the header | +========+=============+===========================================+ | 0x01 | HELLO | Figure 3 | +--------+-------------+-------------------------------------------+ | 0x02 | HELLO_ACK | Figure 4 | +--------+-------------+-------------------------------------------+ | 0x10 | LOAD | Data Descriptor [, Completion Descriptor] | +--------+-------------+-------------------------------------------+ | 0x11 | STORE | Data Descriptor [, Completion Descriptor] | | | (mapped) | [, inline data] | +--------+-------------+-------------------------------------------+ | 0x12 | ALLOC_STORE | Data Descriptor [, Completion Descriptor] | | | (unmapped) | [, inline data] | +--------+-------------+-------------------------------------------+ | 0x13 | FREE | [Completion Descriptor] | +--------+-------------+-------------------------------------------+ | 0x14 | BIND | Ticket (32), Data Length (32), Block | | | | Offset (64) | +--------+-------------+-------------------------------------------+ | 0x20 | COMPLETION | Data Length (32), Reserved (32), Block | | | | Offset (64) | +--------+-------------+-------------------------------------------+ | 0x30 | GOODBYE | none | +--------+-------------+-------------------------------------------+ Table 2: KBTP Opcodes 5.2. Descriptors A Data Descriptor is 16 octets: STag (32 bits), Tagged Offset (64 bits), and Length (32 bits), naming a buffer of the initiator. For LOAD it is the destination; for STORE and ALLOC_STORE it is the source that the target reads with RDMA Read, unless INLINE is set, in which case STag and Tagged Offset are zero, Length is the inline data length, and the data follows the message. A Completion Descriptor is 16 octets: STag (32 bits), Tagged Offset (64 bits), and 32 reserved bits, naming a 24-octet completion record slot of the initiator. Li, et al. Expires 6 May 2027 [Page 12] Internet-Draft KV Cache Block Transfer over RDMA November 2026 6. Operations A target processes the requests of one connection in order. It validates every request before touching memory: the handle MUST be within a range the initiator is authorized for (its own range for stores and frees), lengths MUST NOT exceed Block Size or the initiator's buffer, and, if the request Generation is non-zero, it MUST equal the entry's. A failed check completes the request with the corresponding status and has no other effect. 6.1. Block Load 1. The target reads the table entry of the handle. The entry MUST be VALID, and SEALED unless ALLOW_UNSEALED is set. 2. It RDMA-writes Data Length octets of the block to the Data Descriptor. Data Length MUST NOT exceed the descriptor Length. 3. It completes the request (Section 6.5) with the Block Offset, Data Length, and Generation of the entry. 6.2. Block Mapped Store 1. The target reads the table entry, which MUST be VALID and not SEALED and MUST belong to the initiator's handle range. 2. It obtains the data, from the inline payload or by RDMA Read from the Data Descriptor, and writes it into the block. 3. It updates Data Length and, if SEAL is set, sets SEALED. 4. It completes the request. 6.3. Block Unmapped Store 1. The handle MUST be in the initiator's range and its entry MUST NOT be VALID. 2. The target allocates a block from the ring and marks the slot consumed (Section 3.3). 3. It obtains the data and writes it into the block. 4. It writes the table entry with the Block Offset and Data Length, then with VALID, SEALED if requested, and the next Generation (incremented by one, skipping zero). 5. It completes the request with the Block Offset and Generation. Li, et al. Expires 6 May 2027 [Page 13] Internet-Draft KV Cache Block Transfer over RDMA November 2026 The entry is written after the data, so any initiator that observes VALID in the entry also observes the data. 6.4. Block Free 1. The handle MUST be in the initiator's range and VALID. 2. The target clears VALID and SEALED and increments the Generation (skipping zero) in the table entry. 3. It returns the block to the ring from which it was allocated (Section 3.3), after the entry update is visible. 4. It completes the request. Initiators that may still be loading the block detect the free by the changed Generation. If the block may next be allocated to an initiator of a different trust domain, the target clears its contents before returning it to the ring. 6.5. Completion By default the target completes a request with a COMPLETION message sent as a Send (or Send with Invalidate when INVALIDATE is set). In the RDMAP binding, the contents of a buffer written by RDMA Write are determinate once a subsequent Send has been delivered (rule 5 of Section 5.5 of [RFC5040]), so receipt of the completion implies that the loaded data is in the initiator's buffer. If the target advertised RECORD_COMPLETION and the request has the RECORD flag, the target instead writes a 24-octet completion record to the Completion Descriptor: Request ID (64 bits), Status (16 bits), Generation (16 bits), Data Length (32 bits), and Block Offset (64 bits). A target MUST use completion records only on bindings in which an RDMA Write cannot become visible before the data of earlier RDMA Writes on the same connection. RDMAP does not guarantee this (rules 3 and 5 of Section 5.5 of [RFC5040]), so a target MUST NOT advertise RECORD_COMPLETION in the RDMAP binding. 6.6. Direct Modes Direct modes expose target structures to the initiator and are OPTIONAL. A target MUST offer them only to initiators it trusts for all blocks reachable through the exposed STags. Direct Load (DIRECT_LOAD): through the read-only Table and Pool STags, the initiator RDMA-reads the table entry, then the block, then the entry again, issuing each read only after the previous one has Li, et al. Expires 6 May 2027 [Page 14] Internet-Draft KV Cache Block Transfer over RDMA November 2026 completed. The data is valid if both entry reads show the same Generation with VALID and SEALED set, because a sealed block does not change until it is freed and a free changes the Generation. This costs two round trips but no target processing. Direct Allocation (DIRECT_ALLOC): the initiator performs FetchAdd on its Head with the Add Data and Add Mask of Section 3.3, RDMA-reads the slot at the returned offset (re-reading until the Sequence equals its ticket and the Block Offset is not all ones, and then once more to confirm both values, since a 16-octet read is not guaranteed to be atomic), RDMA-writes the data into the block, and then sends BIND with the ticket, Data Length, and Block Offset. The target verifies that the ticket has not been abandoned, that the slot of the ticket holds that Block Offset, and that the initiator is within its quota; it then marks the slot consumed, writes the table entry, and completes. The initiator never writes ring slots. Frees always use FREE. 7. Status Codes +=======+==================+======================================+ | Value | Name | Meaning | +=======+==================+======================================+ | 0 | OK | Success | +-------+------------------+--------------------------------------+ | 1 | INVALID_HANDLE | Handle outside authorized range | +-------+------------------+--------------------------------------+ | 2 | STALE_GENERATION | Generation mismatch | +-------+------------------+--------------------------------------+ | 3 | NOT_VALID | Entry not bound | +-------+------------------+--------------------------------------+ | 4 | ALREADY_BOUND | Unmapped Store to a bound handle | +-------+------------------+--------------------------------------+ | 5 | NO_SPACE | Quota exceeded or ring empty | +-------+------------------+--------------------------------------+ | 6 | SEALED | Store to a sealed block | +-------+------------------+--------------------------------------+ | 7 | NOT_SEALED | Load of an unsealed block | +-------+------------------+--------------------------------------+ | 8 | TOO_LARGE | Length exceeds block or buffer | +-------+------------------+--------------------------------------+ | 9 | MALFORMED | Request cannot be parsed | +-------+------------------+--------------------------------------+ | 10 | INTERNAL | Target failure | +-------+------------------+--------------------------------------+ | 11 | TICKET_ABANDONED | BIND for a ticket that was reclaimed | +-------+------------------+--------------------------------------+ Li, et al. Expires 6 May 2027 [Page 15] Internet-Draft KV Cache Block Transfer over RDMA November 2026 Table 3: KBTP Status Codes 8. Target Execution Model This section is informative. It shows how the formats allow a target to execute operations without CPU involvement. A target pre-posts, for each request credit, a receive buffer followed by a chain of work requests that its RDMA interface starts when the receive completes. The receive scatter list places the request's Data Descriptor and handle directly into the address fields of the following work requests (hence the ADDR_LE option), and table and ring entries read by the chain are placed into later work requests in the same way. +===========+====================================================+ | Operation | Chain at the target | +===========+====================================================+ | Block | receive request; read table entry into the next | | Load | request; write block to initiator; send completion | +-----------+----------------------------------------------------+ | Block | receive request with inline data; read table | | Mapped | entry; write data to block; write entry; send | | Store | completion | +-----------+----------------------------------------------------+ | Block | receive request with inline data; masked fetch- | | Unmapped | and-add on Head; read slot; check Sequence | | Store | (conditional execution); mark slot consumed; write | | | data to block; write table entry; send completion | +-----------+----------------------------------------------------+ Table 4: Example Work Request Chains The masked FetchAdd of Section 3.3 returns a slot offset that is directly usable as an address, and table entries and slots hold Block Offsets in fixed positions, which is what makes these chains possible. Loads and mapped stores need only data movement. An allocation also needs the Sequence check of Section 3.3 and the consumption of the slot; a chain can perform them only if the interface supports conditional execution of work requests, and otherwise the target handles allocations in software. Frees, which update the Generation and may be deferred, are normally handled in software and are not latency-critical. The target CPU otherwise only refills the pre-posted chains, in batches. Implementations that cannot chain work requests process the same messages in software. 9. Transport Bindings 9.1. RDMAP Binding Li, et al. Expires 6 May 2027 [Page 16] Internet-Draft KV Cache Block Transfer over RDMA November 2026 * KBTP runs over an RDMAP Stream [RFC5040] on DDP [RFC5041] and MPA [RFC5044]. Peers SHOULD use the enhanced connection establishment of [RFC6581]. Targets that advertise DIRECT_ALLOC, and targets that execute allocations in their RDMA interface, MUST support the atomic operations of [RFC7306]. * Requests, HELLO, HELLO_ACK, and COMPLETION are carried in Send or Send with Solicited Event messages; INVALIDATE uses Send with Invalidate. * Loads use RDMA Write; stores without INLINE use RDMA Read by the target. * Ring counters use FetchAdd with the Add Mask of Section 3.3. * Completions are always COMPLETION messages; RECORD_COMPLETION is not offered (Section 6.5). * The security of the binding is that of [RFC5042]; when the network is not trusted, IPsec MUST be used as specified there. 9.2. InfiniBand and RoCEv2 Over an InfiniBand reliable connection or RoCEv2 [IBTA], the same messages are carried with Send, RDMA Write, and RDMA Read. The InfiniBand specification defines an unmasked Fetch and Add; masked variants exist as vendor extensions. With unmasked Fetch and Add, the Head and Tail counters hold the Ticket in their high 32 bits and zero in their low 32 bits (Add Data 0x0000000100000000), and the slot is computed as 16 * (ticket mod N) by the initiator in DIRECT_ALLOC or by the target in software. Completion records on these transports MUST be written with RDMA Write with Immediate, so that the initiator learns of completion from its completion queue rather than by polling memory. 10. Operational Considerations * Block size follows the KV cache block of the serving system (for example 16 tokens per block) or a multiple of it. * Quotas and handle ranges are set at session setup and changed by reconnecting; targets reserve free blocks for the sum of quotas. * Implementations SHOULD expose counters per operation and status, ring occupancy, and quota use per initiator. Li, et al. Expires 6 May 2027 [Page 17] Internet-Draft KV Cache Block Transfer over RDMA November 2026 11. Security Considerations The security considerations of [RFC5042] apply. In the default mode, the target exposes no STag of its own memory to initiators: all access to the block pool, the table, and the ring is performed by the target after validation. The initiator exposes only the buffers named in its descriptors; it SHOULD use STags that are valid for a single request and SHOULD request INVALIDATE so that they are revoked by the completion. Handle ranges prevent an initiator from overwriting or freeing another initiator's blocks. Generations prevent use of stale handles. Quotas bound memory consumption per initiator. Direct modes expose the table and pool for reading, and the ring counters for atomic modification, to the trusted initiator. A misbehaving initiator with DIRECT_ALLOC can take tickets without binding them; the Ticket Timeout bounds how long such blocks are withheld, and quotas bound how many. Targets offer direct modes only to initiators within the same trust domain and verify every BIND against the ring. KV cache blocks contain information derived from user prompts. Connections SHOULD be confined to a trusted network or protected with IPsec, and targets SHOULD clear blocks when they are freed if the blocks may be allocated to initiators of a different trust domain. 12. IANA Considerations IANA is requested to create a registry group named "KV Cache Block Transfer Protocol (KBTP)" with the registries "KBTP Opcodes" (8 bits, initial contents from Table 2), "KBTP Status Codes" (16 bits, initial contents from Table 3), "KBTP Flags" (one entry per opcode and bit of the 16-bit Flags field; initial contents: for LOAD, STORE, ALLOC_STORE, and FREE, the flags of Section 5.1; for GOODBYE, 0x0001 RELEASE), and "KBTP Features" (one entry per bit of the 16-bit Features field, initial contents from Section 4). New values are assigned by Specification Required [RFC8126]; the upper quarter of each value space is reserved for Private Use. The designated experts should check that new opcodes define their validation rules, memory ordering, and completion. This document requests no port number; KBTP is identified by the RDMA connection over which it is established. 13. References 13.1. Normative References Li, et al. Expires 6 May 2027 [Page 18] Internet-Draft KV Cache Block Transfer over RDMA November 2026 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC5040] Recio, R., Metzler, B., Culley, P., Hilland, J., and D. Garcia, "A Remote Direct Memory Access Protocol Specification", RFC 5040, DOI 10.17487/RFC5040, October 2007, . [RFC5041] Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct Data Placement over Reliable Transports", RFC 5041, DOI 10.17487/RFC5041, October 2007, . [RFC5042] Pinkerton, J. and E. Deleganes, "Direct Data Placement Protocol (DDP) / Remote Direct Memory Access Protocol (RDMAP) Security", RFC 5042, DOI 10.17487/RFC5042, October 2007, . [RFC5044] Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, "Marker PDU Aligned Framing for TCP Specification", RFC 5044, DOI 10.17487/RFC5044, October 2007, . [RFC6581] Kanevsky, A., Ed., Bestler, C., Ed., Sharp, R., and S. Wise, "Enhanced Remote Direct Memory Access (RDMA) Connection Establishment", RFC 6581, DOI 10.17487/RFC6581, April 2012, . [RFC7306] Shah, H., Marti, F., Noureddine, W., Eiriksson, A., and R. Sharp, "Remote Direct Memory Access (RDMA) Protocol Extensions", RFC 7306, DOI 10.17487/RFC7306, June 2014, . [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 8126, DOI 10.17487/RFC8126, June 2017, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . 13.2. Informative References Li, et al. Expires 6 May 2027 [Page 19] Internet-Draft KV Cache Block Transfer over RDMA November 2026 [I-D.li-kvcache-efficient-storage] Li, Z., Du, Z., Wang, J., Cheng, W., Zhao, Y., Zhao, C., and X. Sun, "Key-Value (KV) Cache Storage Protocol (KVSP): Indexing, Placement, Prefetch, and Migration for Distributed Inference", Work in Progress, Internet-Draft, draft-li-kvcache-efficient-storage-00, 2 November 2026, . [I-D.talpey-rdma-commit] Talpey, T., "RDMA Extensions for Enhanced Memory Placement", Work in Progress, Internet-Draft, draft- talpey-rdma-commit-02, 25 January 2023, . [IBTA] InfiniBand Trade Association, "InfiniBand Architecture Specification Volume 1, Release 1.7, and Annex A17: RoCEv2", 2023, . [MOONCAKE] Qin, R., Li, Z., He, W., Cui, J., Ren, F., Zhang, M., Wu, Y., Zheng, W., and X. Xu, "Mooncake: Trading More Storage for Less Computation - A KVCache-centric Architecture for Serving LLM Chatbot", 23rd USENIX Conference on File and Storage Technologies (FAST 25), February 2025, . [PAGEDATTENTION] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and I. Stoica, "Efficient Memory Management for Large Language Model Serving with PagedAttention", Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP), October 2023, . [RFC7145] Ko, M. and A. Nezhinsky, "Internet Small Computer System Interface (iSCSI) Extensions for the Remote Direct Memory Access (RDMA) Specification", RFC 7145, DOI 10.17487/RFC7145, April 2014, . [RFC8166] Lever, C., Ed., Simpson, W., and T. Talpey, "Remote Direct Memory Access Transport for Remote Procedure Call Version 1", RFC 8166, DOI 10.17487/RFC8166, June 2017, . Li, et al. Expires 6 May 2027 [Page 20] Internet-Draft KV Cache Block Transfer over RDMA November 2026 Appendix A. Examples ALLOC_STORE of 8 octets of inline data ("KVBLOCK!") to handle 0x4001, Request ID 7, flags INLINE: 01 12 00 01 00 00 00 00 00 00 00 07 00 00 00 00 00 00 40 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 08 4b 56 42 4c 4f 43 4b 21 Its COMPLETION (Generation 5, Status OK, Data Length 8, Block Offset 0x150000, which is block 42 of a pool of 32 KiB blocks): 01 20 00 00 00 00 00 00 00 00 00 07 00 00 00 00 00 00 40 01 00 05 00 00 00 00 00 08 00 00 00 00 00 00 00 00 00 15 00 00 Ring counter for N = 4 (n = 2) with 16-octet slots: Add Data 0x0000000100000010, Add Mask 0x8000000080000020. Successive FetchAdd results on Head, starting from zero, show the ticket advancing while the slot offset wraps after four slots: returned (ticket, offset) new counter value (0, 0) 0x0000000100000010 (1, 16) 0x0000000200000020 (2, 32) 0x0000000300000030 (3, 48) 0x0000000400000000 (4, 0) 0x0000000500000010 A shared ring of four free blocks 100 to 103 (N = F = 4) serves target-executed allocations. The target takes ticket 0 for an ALLOC_STORE of initiator X, and that work request chain is delayed before reading slot 0. Meanwhile ALLOC_STORE requests of initiator Y take tickets 1, 2, and 3 (blocks 101, 102, 103), and Y frees block 101; the free obtains ticket 4, whose slot is slot 0, which has not been consumed yet, so the free is deferred. X's allocation then reads slot 0, obtains block 100, and consumes the slot; the deferred free now writes slot 0 with Sequence 4 and block 101, which the next allocation (ticket 4) obtains. No block is lost or handed out twice. Acknowledgments The authors thank colleagues working on RDMA offload and KV cache storage for their comments. Authors' Addresses Li, et al. Expires 6 May 2027 [Page 21] Internet-Draft KV Cache Block Transfer over RDMA November 2026 Zhiqiang Li China Mobile Beijing 100053 China Email: lizhiqiangyjy@chinamobile.com Zongpeng Du China Mobile Beijing 100053 China Email: duzongpeng@chinamobile.com Junjie Wang Centec Shanghai 201203 China Email: wangjj@centec.com Wei Cheng Centec Shanghai 201203 China Email: chengw@centec.com Ying Zhao Centec Shanghai 200000 China Email: zhaoying@centec.com Chunhao Zhao SAIA Shanghai 200125 China Email: chunhao.zhao@sh-aia.com Li, et al. Expires 6 May 2027 [Page 22] Internet-Draft KV Cache Block Transfer over RDMA November 2026 Xun Sun Inesa Shanghai 200030 China Email: sunxun@inesa.com Li, et al. Expires 6 May 2027 [Page 23]