Network Working Group Z. Li Internet-Draft Z. Du Intended status: Standards Track China Mobile Expires: 6 May 2027 J. Wang W. Cheng Y. Zhao Centec C. Zhao SAIA X. Sun Inesa 2 November 2026 Key-Value (KV) Cache Storage Protocol (KVSP): Indexing, Placement, Prefetch, and Migration for Distributed Inference draft-li-kvcache-efficient-storage-00 Abstract Large language model inference keeps the attention key-value (KV) cache of processed tokens so that they are not recomputed. When inference instances, cache tiers, and schedulers are separate components, possibly from different implementers, they need a common way to name reusable KV cache blocks, find where copies reside, protect copies from eviction while they are being loaded, steer work to the instance where loading is cheapest, and move a session to another instance after a failure. This document specifies the KV Cache Storage Protocol (KVSP), an HTTP-based protocol that defines a compatibility descriptor, keyed block identifiers that bind a block to its entire token prefix, a self-describing block container, a replica index with layer-granular residency, instance status reporting, leased prefetch directives, session checkpoints for migration, and a resharding procedure that retains sequence shards in place when a parallel group is resized. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Li, et al. Expires 6 May 2027 [Page 1] Internet-Draft KV Cache Storage Protocol November 2026 Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 6 May 2027. Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Problem Statement . . . . . . . . . . . . . . . . . . . . 3 1.2. Protocol Overview . . . . . . . . . . . . . . . . . . . . 4 1.3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.4. Relationship to Other Work . . . . . . . . . . . . . . . 5 2. Conventions and Definitions . . . . . . . . . . . . . . . . . 5 3. Architecture . . . . . . . . . . . . . . . . . . . . . . . . 6 4. Data Model . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.1. Compatibility Descriptor and Compatibility ID . . . . . . 7 4.2. Scopes and Block Keys . . . . . . . . . . . . . . . . . . 8 4.3. KV Block Container . . . . . . . . . . . . . . . . . . . 8 4.4. Records . . . . . . . . . . . . . . . . . . . . . . . . . 9 5. HTTP Usage . . . . . . . . . . . . . . . . . . . . . . . . . 11 6. Procedures . . . . . . . . . . . . . . . . . . . . . . . . . 13 6.1. Publishing and Withdrawing Replicas . . . . . . . . . . . 13 6.2. Lookup . . . . . . . . . . . . . . . . . . . . . . . . . 13 6.3. Instance Status and Lifecycle . . . . . . . . . . . . . . 14 6.4. Placement Inputs . . . . . . . . . . . . . . . . . . . . 14 6.5. Prefetch Directives . . . . . . . . . . . . . . . . . . . 15 6.6. Leases . . . . . . . . . . . . . . . . . . . . . . . . . 16 6.7. Block Retrieval and Storage . . . . . . . . . . . . . . . 16 6.8. Session Checkpoints and Migration . . . . . . . . . . . . 16 6.9. Sequence-Sharded Caches and Resharding . . . . . . . . . 17 7. Error Handling . . . . . . . . . . . . . . . . . . . . . . . 19 8. Operational Considerations . . . . . . . . . . . . . . . . . 19 Li, et al. Expires 6 May 2027 [Page 2] Internet-Draft KV Cache Storage Protocol November 2026 9. Security Considerations . . . . . . . . . . . . . . . . . . . 20 10. Privacy Considerations . . . . . . . . . . . . . . . . . . . 21 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 11.1. Media Types . . . . . . . . . . . . . . . . . . . . . . 21 11.2. KVSP Registries . . . . . . . . . . . . . . . . . . . . 22 11.2.1. KVSP Problem Types . . . . . . . . . . . . . . . . . 22 11.2.2. KVSP Codecs . . . . . . . . . . . . . . . . . . . . 22 11.2.3. KVSP Element Types and Positional Encodings . . . . 22 11.2.4. KVSP Storage Tiers . . . . . . . . . . . . . . . . . 22 11.2.5. KVSP Instance States . . . . . . . . . . . . . . . . 23 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 23 12.1. Normative References . . . . . . . . . . . . . . . . . . 23 12.2. Informative References . . . . . . . . . . . . . . . . . 24 Appendix A. Test Vectors . . . . . . . . . . . . . . . . . . . . 26 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 26 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 26 1. Introduction 1.1. Problem Statement The KV cache of a transformer model holds, for every processed token and every layer, the key and value vectors used by attention. It lets each decoding step reuse earlier computation, but it imposes three costs on serving systems: * Footprint: the cache grows linearly with context length and batch size and quickly exhausts accelerator memory, which limits both concurrency and maximum context length. * Request-bound lifetime: when the cache is discarded at the end of each request, requests that share a long system prompt or document prefix recompute the same state, which increases the time to first token. * Swap cost: moving cache contents between accelerator memory, host memory, and storage is slow when done synchronously and stalls computation. Serving systems address these costs by keeping caches beyond the lifetime of a request, storing them in several tiers, and reusing them across requests and instances [PAGEDATTENTION] [MOONCAKE]. These systems are currently built as single-vendor stacks. When the scheduler, the inference engines, and the cache tiers come from different implementers, or span administrative boundaries within an operator, every pair of components needs agreement on the same questions: which caches are interchangeable, what a block is called, where its copies are, how long a copy will stay, which instance Li, et al. Expires 6 May 2027 [Page 3] Internet-Draft KV Cache Storage Protocol November 2026 should serve the next request, and how to continue a session when an instance fails. [I-D.li-cats-kv-cache-distribution] states these needs as requirements. This document specifies a protocol that meets them. 1.2. Protocol Overview KVSP defines: * a Compatibility Descriptor and its digest, the Compatibility ID, which state when two caches can substitute for each other (Section 4.1); * Block Keys computed as a keyed hash chain over token identifiers, so that a key names a block together with its entire prefix and cannot be inverted by parties that lack the scope key (Section 4.2); * a KV Block Container format that supports retrieval of a range of layers (Section 4.3); * an Index Service that records replicas, including which layers of a block reside in which storage tier (Section 6.1); * instance status reports and an instance lifecycle including draining and sleeping (Section 6.3); * prefetch directives with eviction-protecting leases, which let a placement controller load caches into an instance before the request arrives (Section 6.5); * session checkpoints that let another instance resume a session after a failure (Section 6.8); and * a resharding procedure for caches split by token range across a parallel group, which keeps shards in place when they stay on the same instance (Section 6.9). 1.3. Scope KVSP standardizes the information exchanged between components and the obligations attached to it. It does not standardize attention kernels, memory allocators, eviction policies, or the scheduling policy of the placement controller; it standardizes the inputs that such a policy consumes and the directives it issues. Inference requests themselves (for example, chat completion APIs) are out of scope. Bulk block transfer uses HTTP by default; other transports such as Media over QUIC Transport Li, et al. Expires 6 May 2027 [Page 4] Internet-Draft KV Cache Storage Protocol November 2026 [I-D.li-kvcache-efficient-transmission] or RDMA [I-D.li-rdma-kvcache-transfer-optimization] can be used where available. 1.4. Relationship to Other Work Computing-Aware Traffic Steering (CATS) [I-D.ietf-cats-framework] steers requests to service instances using computing and network metrics. The instance status and lookup estimates of KVSP can serve as additional inputs to such steering, as described in [I-D.li-cats-kv-cache-distribution]. Naming Things with Hashes [RFC6920] defines hash-based identifiers; KVSP uses a keyed construction instead, because plain hashes of short token sequences can be inverted by enumeration (Section 10). 2. Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. Token: An integer identifier produced by the model's tokenizer. Block: The KV cache entries of B consecutive tokens, for all layers, where B is the block size of the Compatibility Descriptor. Block i covers token positions [i*B, (i+1)*B). Scope: A set of principals that are allowed to share caches, such as a tenant. A scope is identified by a Scope ID and holds a secret Scope Key. Holder: A component that stores blocks and serves them, for example an inference instance (for its accelerator and host memory) or a cache server. Tier: A class of storage at a holder, such as accelerator memory or host memory (Section 11.2.4). Replica: A copy of a block, or of some of its layers, at a holder. Index Service (IS): The component that records replicas and answers lookups. Placement Controller (PC): The component that assigns requests to inference instances and issues directives to them. It is often called a global scheduler. Li, et al. Expires 6 May 2027 [Page 5] Internet-Draft KV Cache Storage Protocol November 2026 Inference Instance (II): A model execution engine with its own accelerator memory. Session: One generation task (a request and its output tokens) whose state may outlive the instance that started it. Lease: A holder's commitment not to evict specified replicas until the lease expires or is released. Parallel Group: A set of instances that jointly hold the cache of one context, split by token range (sequence parallelism). JSON [RFC8259] member names in this document are case-sensitive. "hex32" denotes 32 octets encoded as 64 lowercase hexadecimal characters; "hex16" denotes 16 octets encoded as 32 lowercase hexadecimal characters. Durations are integers in milliseconds unless the member name says otherwise. 3. Architecture +----------------------+ status, | Placement Controller | prefetch, resume, checkpoints -->| (PC) |-->reshard directives +----------+-----------+ | lookup v +----------------------+ publish, --->| Index Service (IS) | withdraw +----------------------+ +---------------+ +---------------+ +---------------+ | Inference | | Inference | | Cache server | | Instance 1 |<--->| Instance 2 |<--->| (holder) | | tiers: device,| | tiers: device | | tiers: host, | | host | | | | storage | +---------------+ +---------------+ +---------------+ block GET, PUT, and leases between all holders Figure 1: KVSP Roles Roles can be combined. A deployment can run the PC and IS as one service, and every inference instance is normally also a holder for its own tiers. All roles are reached through HTTPS base URIs configured by the operator (Section 5). Li, et al. Expires 6 May 2027 [Page 6] Internet-Draft KV Cache Storage Protocol November 2026 4. Data Model 4.1. Compatibility Descriptor and Compatibility ID A KV cache produced under one configuration is only valid for another computation if the model weights, the tokenizer, and every parameter that affects the stored tensors are identical. The Compatibility Descriptor is a JSON object with the following members, all REQUIRED unless stated otherwise: model: A string naming the model. It is part of the identity, so all parties MUST use the same string for the same model. model_digest: "sha-256:" followed by the lowercase hex SHA-256 [RFC6234] of the model weights, computed over the weight files concatenated in ascending byte order of their file names. tokenizer_digest: "sha-256:" followed by the lowercase hex SHA-256 of the tokenizer definition file. adapters: OPTIONAL array of adapter digests, in the same form as model_digest, in the order of application. The member MUST be absent when no adapter is applied; an empty array is not allowed. layers, kv_heads, head_dim: Integers describing the cache geometry. kv_dtype: The element type of the stored cache, from the "KVSP Element Types" registry (Section 11.2.3). positional: The positional encoding variant, from the "KVSP Positional Encodings" registry, optionally followed by ";" and parameters in the registered syntax. Parameters that are absent take the defaults of the model, and a descriptor MUST include them whenever the model is run with non-default values. block_tokens: The block size B in tokens. layout: The layout identifier for codec "raw" (Section 4.3); this document defines "kvsp-layout-1". The Compatibility ID is: compat_id = SHA-256( "KVSP-COMPAT-1" || 0x00 || JCS(descriptor) ) Li, et al. Expires 6 May 2027 [Page 7] Internet-Draft KV Cache Storage Protocol November 2026 where JCS is the JSON Canonicalization Scheme [RFC8785] and || denotes concatenation. The Compatibility ID is carried as hex32. Two caches MUST NOT be substituted for each other unless their Compatibility IDs are equal. Implementations MUST compute the Compatibility ID from the descriptor rather than accept it from a peer without verification whenever the descriptor is available. 4.2. Scopes and Block Keys Each scope has a Scope Key of at least 32 random octets, shared by the principals of the scope through means outside this document. The Scope ID is: scope_id = first 16 octets of SHA-256( "KVSP-SCOPE-1" || 0x00 || scope_key ) The Block Key of block i of a token sequence t[0], t[1], ... is defined recursively, with K[-1] equal to 32 zero octets: K[i] = HMAC-SHA-256( scope_key, "KVSP-BLOCK-1" || 0x00 || compat_id || K[i-1] || uint32(t[i*B]) || ... || uint32(t[i*B+B-1]) ) HMAC is defined in [RFC2104]. compat_id is the 32 raw octets, and uint32() is the token in four octets in network byte order. Because K[i] covers K[i-1], a Block Key identifies the whole prefix [0, (i+1)*B) and not only the tokens of block i. Only complete blocks have keys; the tokens after the last complete block are not shared. Block Keys are carried as hex32. A deployment that wants to share a public prefix (for example, a common system prompt) across tenants creates a dedicated shared scope for it. Caches MUST NOT be shared across scopes otherwise. 4.3. KV Block Container Blocks are exchanged in the KV Block Container. A container holds a contiguous range of layers of one block: Li, et al. Expires 6 May 2027 [Page 8] Internet-Draft KV Cache Storage Protocol November 2026 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Magic ("KVB1" = 0x4B564231) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Version | Flags | Codec | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | First Layer | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Layer Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Token Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Layer Count entries of { K Length (64), V Length (64) } | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data: for each layer in ascending order, K octets, V octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 2: KV Block Container Header Version is 1. Flags are reserved and set to zero. Codec is a value from the "KVSP Codecs" registry (Section 11.2.2). Integers are in network byte order. The ascending layer order allows a receiver to start using layer l as soon as it has been received, overlapping computation with loading. For codec 0x0001 ("raw"), the K and V sections of a layer each hold Token Count x kv_heads x head_dim elements of kv_dtype, in row-major order with the token index varying slowest and the dimension index fastest, each element in little-endian octet order; this is layout "kvsp-layout-1". Other codecs (for example, quantized or entropy- coded caches) define their own section encoding. The layer digest of layer l is the SHA-256 of its K section followed by its V section. The digest of a replica is the SHA-256 of the concatenation of the layer digests of all layers, in ascending order. This construction lets a client that retrieves a range of layers verify each of them against the layer digests listed in the Replica Record, and verify that list against the replica digest. 4.4. Records The following JSON objects are used in requests and responses. Unknown members MUST be ignored. Replica Record: Li, et al. Expires 6 May 2027 [Page 9] Internet-Draft KV Cache Storage Protocol November 2026 { "block_key": "", "compat_id": "", "scope_id": "", "holder": "https://h1.example/kvsp/", "codec": 1, "size": 2097152, "digest": "sha-256:", "layer_digests": ["", "", "..."], "residency": [ { "tier": "device", "layers": [0, 15] }, { "tier": "host", "layers": [16, 31], "read_mbit_per_s": 400000 } ], "ttl_ms": 600000 } "layer_digests" lists the layer digests (Section 4.3) in layer order; it is REQUIRED for replicas that are served by layer range. The residency array states which layers reside in which tier. Layer ranges are inclusive and MUST NOT overlap. The optional "read_mbit_per_s" of a residency entry states the rate at which the holder can read that tier. A holder that keeps some layers of a block in accelerator memory and the others in host memory, to overlap layer loading with computation, reports this split so that placement can account for it. A replica can also be served through Media over QUIC Transport [I-D.li-kvcache-efficient-transmission]. Its record then has a "holder" that is a "moqt" URI of the publishing endpoint or relay and a member "moqt_namespace" holding the base64url encoding (without padding) of the five Track Namespace Fields, each preceded by its length as one octet. The publisher of that namespace is the holder for the purposes of authorization, and the digests have the meaning defined in this section, which the MOQT object digests reproduce for codec "raw". Instance Status Record: Li, et al. Expires 6 May 2027 [Page 10] Internet-Draft KV Cache Storage Protocol November 2026 { "instance": "https://ii7.example/kvsp/", "epoch": 12, "state": "active", "running": 18, "queued": 3, "batch_size": 32, "kv_utilization": 0.71, "prefill_tokens_per_s": 42000, "decode_tokens_per_s": 3100, "links": [ { "holder": "https://h1.example/kvsp/", "mbit_per_s": 200000, "rtt_us": 12 } ], "requested_state": null, "ttl_ms": 3000 } Session Checkpoint: { "session": "6f1d0a9e-2c55-4d7e-9a0e-5b7f2f9f6a11", "epoch": 1, "compat_id": "", "scope_id": "", "prompt_blocks": ["", ""], "prompt_tail": [374, 264], "generated": [1296, 315, 279], "output_offset": 3, "sampler": { "method": "top-p", "params": {"p": 0.9}, "seed": 7, "step": 3, "deterministic": false } } The session identifier is a UUID [RFC9562]. output_offset is the number of output tokens covered by the checkpoint. prompt_tail holds the prompt tokens after the last complete prompt block. "sampler.deterministic" is true only if every instance eligible to resume the session produces bit-identical output tokens from the same state, for example greedy decoding with batch-invariant kernels on identical software and hardware. 5. HTTP Usage KVSP is an HTTP application in the sense of BCP 56 [RFC9205]. Each role is reached through a base URI configured by the operator; KVSP defines no well-known URI and no fixed path prefix outside that base. The resource paths in Table 1 are relative to the base URI of the role. Clients MUST use HTTPS with TLS 1.3 [RFC8446] or later, and both sides MUST authenticate with certificates. HTTP/2 [RFC9113] or HTTP/3 [RFC9114] SHOULD be used. Li, et al. Expires 6 May 2027 [Page 11] Internet-Draft KV Cache Storage Protocol November 2026 Request and response bodies, other than block data, use the media type "application/kvsp+json". Errors are reported with Problem Details [RFC9457] using the types defined in Section 7. Clients MUST handle status codes not listed in this document according to their class [RFC9110]. Responses other than block data MUST carry "Cache- Control: no-store". KVSP defines no new HTTP fields; request parameters are carried in the request body, in the path below the base URI, or in the query component, which BCP 56 permits for resources under the application's control. +========+=====================================+================+ | Role | Method and Path | Purpose | +========+=====================================+================+ | IS | POST lookup | Longest-prefix | | | | lookup | | | | (Section 6.2) | +--------+-------------------------------------+----------------+ | IS | PUT, DELETE replicas/{holder- | Publish, | | | id}/{block_key} | withdraw | | | | (Section 6.1) | +--------+-------------------------------------+----------------+ | Holder | GET, PUT | Retrieve, | | | blocks/{compat}/{scope}/{block_key} | store | | | | (Section 6.7) | +--------+-------------------------------------+----------------+ | Holder | POST leases; PUT, DELETE | Leases | | | leases/{id} | (Section 6.6) | +--------+-------------------------------------+----------------+ | PC | PUT instances/{instance-id}/status | Status | | | | (Section 6.3) | +--------+-------------------------------------+----------------+ | PC | PUT sessions/{session}/checkpoint | Checkpoint | | | | (Section 6.8) | +--------+-------------------------------------+----------------+ | II | POST prefetch; GET prefetch/{id} | Prefetch | | | | (Section 6.5) | +--------+-------------------------------------+----------------+ | II | POST sessions/{session}/resume | Migration | | | | (Section 6.8) | +--------+-------------------------------------+----------------+ | II | POST reshard | Resharding | | | | (Section 6.9) | +--------+-------------------------------------+----------------+ | II | PUT state | Lifecycle | | | | (Section 6.3) | +--------+-------------------------------------+----------------+ Table 1: KVSP Resources Li, et al. Expires 6 May 2027 [Page 12] Internet-Draft KV Cache Storage Protocol November 2026 {holder-id} and {instance-id} are the base64url encoding without padding [RFC4648] of the SHA-256 of the holder's or instance's base URI, which gives each a fixed-length path-safe identifier. 6. Procedures 6.1. Publishing and Withdrawing Replicas A holder publishes each replica it is willing to serve with PUT, carrying a Replica Record. The Index Service MUST verify that the authenticated client is the holder named in the record or is authorized to publish on its behalf, and that the client is authorized for the record's scope. A replica record expires after ttl_ms; the IS caps ttl_ms at a configured maximum. Holders refresh records before expiry with the same PUT, which is idempotent. A holder MUST withdraw a record with DELETE before or immediately after evicting the replica, and MUST NOT evict a replica covered by an unexpired lease (Section 6.6). The index is advisory: a holder can evict an unleased replica before the withdrawal reaches the IS. Clients therefore treat a 404 from a holder as a cache miss and continue with the next replica or with recomputation. 6.2. Lookup A client (normally the PC) computes the Block Keys of the prompt and sends them, in prefix order, to the IS: POST {is-base}lookup { "scope_id": "", "compat_id": "", "block_keys": ["", "", ""], "targets": ["https://ii7.example/kvsp/", "https://ii9.example/kvsp/"] } The IS returns "matched", the number of leading keys for which at least one unexpired replica exists, and for each matched key the replica records. Because Block Keys are chained, the IS MUST stop at the first key without a replica. When "targets" is present and the IS has the needed status and link information, it also returns, for each target instance, "fetch_us" (the estimated time to make the matched blocks resident at that instance, Section 6.4) and "recompute_us" (the estimated time to prefill the same tokens on that instance). The IS MUST NOT return replicas of a scope for which the client is not authorized. Li, et al. Expires 6 May 2027 [Page 13] Internet-Draft KV Cache Storage Protocol November 2026 6.3. Instance Status and Lifecycle Each inference instance sends its Instance Status Record to the PC at an interval no longer than one third of its ttl_ms. The PC considers an instance failed when its record expires. The "epoch" of an instance MUST increase every time it restarts; the start time of the instance in milliseconds since the Unix epoch satisfies this without persistent state, provided that the instance's clock does not move backward across restarts; otherwise the instance persists a counter. The PC MUST discard records with an epoch lower than the highest it has seen for that instance. Instances move between the states active, draining, sleeping, and failed. An instance whose load has stayed below a configured threshold for a configured time MAY ask to sleep by setting "requested_state" to "sleeping". The PC answers the status PUT with 200 and a body {"state": "draining"} to accept, or {"state": "active"} to refuse. The PC can also change the state of an instance with PUT state on the instance. * The PC MUST NOT assign new requests or send prefetch directives to an instance that is not active. * A draining instance MUST complete its in-flight sessions and MUST NOT accept new ones; when it has none left it enters sleeping and reports it. * A sleeping instance becomes active only on a state change from the PC. 6.4. Placement Inputs The PC chooses an instance for each request. The policy is local, but it is expected to compare, for each candidate, the cost of loading cached blocks with the cost of recomputing them. The following estimates SHOULD be used, by the IS when it answers a lookup with targets and by the PC otherwise: fetch_us(i) = max over sources s of ( rtt_us(i,s) + 8 * bytes_from(s) / mbit_per_s(i,s) ) for blocks not already resident at i recompute_us(i) = 1e6 * tokens / prefill_tokens_per_s(i) wait_us(i) = queued(i) * mean service time at i cost(i) = wait_us(i) + min( fetch_us(i) + recompute_us_of_unmatched(i), recompute_us_of_all(i) ) Li, et al. Expires 6 May 2027 [Page 14] Internet-Draft KV Cache Storage Protocol November 2026 where bytes_from(s) is the volume the instance would load from source s, and mbit_per_s(i,s) is the link rate reported by instance i for source s, reduced, for layers that the source holds in a storage tier, to the "read_mbit_per_s" of that tier in the Replica Record. The PC SHOULD assign the request to an active instance with the lowest cost. When no active instance has acceptable cost or capacity, the PC MAY start a new instance and SHOULD send it a prefetch directive at once, so that cache loading overlaps the instance start. 6.5. Prefetch Directives To load caches into an instance before a request reaches it, the PC sends a prefetch directive: POST {ii-base}prefetch { "directive": "a2b7c9e0-1f3d-4c1a-8f5b-6d2e9a7c3b10", "request_ref": "opaque-value-from-PC", "scope_id": "", "compat_id": "", "blocks": [ { "block_key": "", "sources": ["https://h1.example/kvsp/"] } ], "layer_order": "ascending", "deadline_ms": 50, "lease_ms": 2000 } The instance answers 202 (Accepted) with a Location field naming the directive's status resource, and processes the directive while it continues to execute current work; prefetch MUST NOT block execution of requests already admitted. For each block it does not already hold, the instance acquires a lease of lease_ms at one of the listed sources (Section 6.6), so that the source cannot evict the block while it is being loaded, retrieves the block (Section 6.7), verifies the digest, places it in its tiers, and publishes its new replica (Section 6.1). The state of the directive ("pending", "partial", "complete", or "failed", with the list of loaded keys) can be read with GET. If the deadline expires, the instance stops loading and reports "partial"; the request is then served with the loaded prefix and the remaining tokens are recomputed. "request_ref" is an opaque value that lets the PC associate the directive with the inference request it will route; how the request carries the value is outside the scope of this document. Li, et al. Expires 6 May 2027 [Page 15] Internet-Draft KV Cache Storage Protocol November 2026 6.6. Leases POST leases on a holder, with a body listing block keys and "ttl_ms", creates a lease and returns its identifier and the granted TTL. PUT on the lease renews it; DELETE releases it. While a lease is valid the holder MUST NOT evict the leased replicas. A holder MUST enforce a per-client quota on leased octets and lease TTL and returns the problem type "kvsp-lease-quota" when a request exceeds it. A holder that must evict leased data due to a fault MUST first withdraw the affected replica records. 6.7. Block Retrieval and Storage Block resources are named blocks/{compat}/{scope}/{block_key}, where {compat} is the Compatibility ID (hex32), {scope} the Scope ID (hex16), and {block_key} the Block Key (hex32), so that retrieval and storage address the same resource. GET on it returns the container (Section 4.3) with media type "application/kvsp-block". The query parameter "layers=a-b" requests the inclusive layer range a..b; a holder that does not hold those layers returns 404 with problem type "kvsp-layers-unavailable". The query parameter "codec=n" requests a codec; a holder MAY transcode. Holders MUST include a "Repr-Digest" field [RFC9530] with the SHA-256 of the returned container. Block representations are immutable but confidential; holders SHOULD send "Cache-Control: private, immutable" with a max-age no longer than the remaining TTL of the replica. Clients MUST verify Repr-Digest. When the returned codec equals the replica's codec, clients MUST also verify the layer digests of the returned layers against the Replica Record, and, when all layers were retrieved, the replica digest. Transcoded representations are protected only by Repr-Digest, so clients that need producer-level integrity request the replica's codec. PUT on the block resource with "If-None-Match: *" stores a block at a holder. The holder MUST reject the request if the client is not authorized for the scope. The key identifies the token prefix, not the stored octets, so a holder that already has a replica for the key returns 412 and keeps its copy. 6.8. Session Checkpoints and Migration An instance that serves a session SHOULD publish a Session Checkpoint to the PC with PUT sessions/{session}/checkpoint whenever a block of generated tokens completes, and at least every 64 output tokens. It also publishes replicas for the generated blocks, so that their KV cache survives the instance. The PC records the checkpoint with the highest output_offset for the current session epoch and MUST reject a Li, et al. Expires 6 May 2027 [Page 16] Internet-Draft KV Cache Storage Protocol November 2026 checkpoint with a lower epoch with problem type "kvsp-stale-epoch". Output continuity depends on the sampler. If the checkpoint states that the sampler is deterministic, a resumed instance regenerates exactly the tokens the failed instance produced after the last checkpoint, and the front end discards the duplicates by index. Otherwise the instance MUST NOT release an output token to the front end before the PC has acknowledged a checkpoint that covers it; it can checkpoint several tokens at a time to bound the added latency. Every output token carries its index and the session epoch, and front ends MUST discard tokens that carry a lower epoch than one they have already seen for the session. When the instance fails, the PC selects a new instance as in Section 6.4, increments the session epoch, and sends POST sessions/{session}/resume to it with the latest checkpoint and the new epoch. The new instance: 1. loads the blocks listed in the checkpoint and the generated blocks published by the failed instance; 2. recomputes the KV cache of prompt_tail and of generated tokens not covered by loaded blocks; 3. restores the sampler with the given method, parameters, seed, and step; and 4. continues generation and labels each output token with its index, starting at output_offset, so that the front end delivers every index exactly once. Any message from the failed instance carrying the old epoch MUST be rejected. Under these rules the output already delivered is never repeated or altered, and the client receives a single consistent sample. 6.9. Sequence-Sharded Caches and Resharding For long contexts, a parallel group can split one context by token range across several instances. The PC describes the split in a Shard Map: Li, et al. Expires 6 May 2027 [Page 17] Internet-Draft KV Cache Storage Protocol November 2026 { "map": "c3f8...e1", "version": 4, "shards": [ { "tokens": [0, 32768], "instance": "https://ii1.example/kvsp/" }, { "tokens": [32768, 65536], "instance": "https://ii2.example/kvsp/" }, { "tokens": [65536, 98304], "instance": "https://ii3.example/kvsp/" } ] } Token ranges are half-open, contiguous, and non-overlapping. The computing demand of a context changes between phases: prefill is compute-bound and benefits from many instances, while early decoding steps need fewer instances and later ones need more memory. The PC can therefore change the group between iterations. To move from version v to v+1, it sends each instance in either version a reshard plan with POST reshard: { "map": "c3f8...e1", "from": 4, "to": 5, "deadline_ms": 200, "retain": [[0, 32768]], "fetch": [{ "tokens": [32768, 49152], "from": "https://ii2.example/kvsp/" }], "release": [] } The following rules apply: * A token range assigned to the same instance in both versions MUST appear in "retain" for that instance and MUST NOT be transferred. Retained ranges keep their memory in place, so a change of parallelism costs no transfer for them. * An instance acknowledges the plan with 200 after it has loaded all ranges listed in "fetch". * Version v remains authoritative until all instances have acknowledged; the PC then sends POST reshard with {"map": ..., "commit": v+1} to all members, after which instances release the ranges listed in "release". * If any instance fails to acknowledge within deadline_ms, the PC sends POST reshard with {"map": ..., "abort": v+1}, and the members discard the ranges they fetched for version v+1. Li, et al. Expires 6 May 2027 [Page 18] Internet-Draft KV Cache Storage Protocol November 2026 7. Error Handling +=========================+========+=============================+ | Type suffix | Status | Meaning | +=========================+========+=============================+ | kvsp-not-found | 404 | Block, lease, session, or | | | | directive unknown | +-------------------------+--------+-----------------------------+ | kvsp-scope-denied | 403 | Client not authorized for | | | | the scope | +-------------------------+--------+-----------------------------+ | kvsp-incompatible | 409 | Compatibility ID mismatch | +-------------------------+--------+-----------------------------+ | kvsp-lease-quota | 429 | Lease quota exceeded | +-------------------------+--------+-----------------------------+ | kvsp-stale-epoch | 409 | Epoch older than current | +-------------------------+--------+-----------------------------+ | kvsp-invalid-state | 409 | Operation not allowed in | | | | the instance state | +-------------------------+--------+-----------------------------+ | kvsp-digest-mismatch | 422 | Stored data does not match | | | | its digest | +-------------------------+--------+-----------------------------+ | kvsp-plan-conflict | 409 | Reshard plan does not match | | | | the current map version | +-------------------------+--------+-----------------------------+ | kvsp-layers-unavailable | 404 | Requested layers not held | +-------------------------+--------+-----------------------------+ Table 2: KVSP Problem Types The type URI of each problem is "https://iana.org/assignments/kvsp#" followed by the type suffix. These types are specific to KVSP and are therefore registered in a KVSP registry rather than in the "HTTP Problem Types" registry, which accepts only widely applicable types (Section 4.2 of [RFC9457]). A client that receives a block whose digest does not match MUST discard it, SHOULD report it to the IS by withdrawing the replica on the holder's behalf if authorized, and continues with another replica or recomputation. 8. Operational Considerations * Replica TTLs trade index accuracy against update load; a TTL of a few minutes with refresh at one third of the TTL is a reasonable start. * KVSP uses durations rather than timestamps, so it does not depend on synchronized clocks. Li, et al. Expires 6 May 2027 [Page 19] Internet-Draft KV Cache Storage Protocol November 2026 * The index can be partitioned by Block Key, since keys are uniformly distributed. * Deployment can be incremental: instances that do not implement KVSP simply recompute; holders can be added without changes to inference engines that already publish their tiers. * Implementations SHOULD report lookup hit ratio, fetched versus recomputed tokens, lease rejections, digest failures, and migrations. 9. Security Considerations All KVSP exchanges are authenticated with mutual TLS. Each role MUST authorize requests by the authenticated identity: only holders publish replicas for themselves, only the PC sends directives to instances and records checkpoints, and access to blocks requires authorization for the block's scope. Cache poisoning. A malicious holder could serve incorrect data for a valid key, which would silently corrupt model output. Digests in replica records and Repr-Digest detect corruption in transit and at honest holders but not a holder that publishes a wrong digest. Deployments MUST therefore restrict publication and service of blocks to holders trusted for the scope. Where stronger assurance is needed, the producing instance can publish the digest itself, and consumers accept only replicas whose digest matches that of the producer. Cross-tenant inference. Whether a cache lookup hits, or how fast a request is served, reveals whether another principal processed the same prefix. Scopes prevent such disclosure across tenants because keys of different scopes are unrelated and lookups are confined to authorized scopes. Within a scope, any principal can test whether another principal of the same scope has processed a guessed prefix, by lookup or by timing; scopes therefore SHOULD contain only principals that may learn each other's prompts, and shared scopes SHOULD be used only for content that is public within the sharing group. Denial of service. Leases reserve holder memory. Quotas on leased octets, lease TTL, the number of pending directives, and lookup size are REQUIRED. Directives can make an instance load data; instances MUST accept them only from the PC. Session takeover. Session epochs fence a failed or partitioned instance, so that only one instance can extend a session. Li, et al. Expires 6 May 2027 [Page 20] Internet-Draft KV Cache Storage Protocol November 2026 10. Privacy Considerations KV caches are derived from user prompts and outputs, and model inversion techniques can recover input text from cached keys and values. KV cache data, checkpoints (which contain output tokens), and replica records MUST be treated with the same confidentiality as the prompts and outputs themselves [RFC6973]. Holders SHOULD encrypt stored blocks and SHOULD apply the retention limits of the originating service. Plain hashes of token sequences can be inverted by enumerating likely prompts. Block Keys are keyed with the Scope Key, so the IS and other parties without the key cannot test guesses. The Scope Key MUST NOT be disclosed to the IS. 11. IANA Considerations 11.1. Media Types IANA is requested to register the following media types in the "Media Types" registry [RFC6838]. Type name: application; Subtype name: kvsp+json; Required parameters: N/A; Optional parameters: N/A; Encoding considerations: binary (UTF-8 JSON); Security considerations: see Section 9 of RFC XXXX; Interoperability considerations: none; Published specification: RFC XXXX; Applications that use this media type: KV cache storage and distributed inference systems; Fragment identifier considerations: none; Additional information: Deprecated alias names for this type: N/A, Magic number(s): N/A, File extension(s): N/A, Macintosh file type code(s): N/A; Person and email address to contact: IETF ; Intended usage: COMMON; Restrictions on usage: none; Author: IETF; Change controller: IETF. Type name: application; Subtype name: kvsp-block; Required parameters: N/A; Optional parameters: N/A; Encoding considerations: binary; Security considerations: see Section 9 of RFC XXXX; Interoperability considerations: none; Published specification: Section 4.3 of RFC XXXX; Applications that use this media type: KV cache storage and distributed inference systems; Fragment identifier considerations: none; Additional information: Deprecated alias names for this type: N/A, Magic number(s): 0x4B564231 at offset 0, File extension(s): N/A, Macintosh file type code(s): N/A; Person and email address to contact: IETF ; Intended usage: COMMON; Restrictions on usage: none; Author: IETF; Change controller: IETF. Li, et al. Expires 6 May 2027 [Page 21] Internet-Draft KV Cache Storage Protocol November 2026 11.2. KVSP Registries IANA is requested to create a registry group named "KV Cache Storage Protocol (KVSP)" at "https://www.iana.org/assignments/kvsp" with the following registries. New assignments use the Specification Required policy [RFC8126]. The designated experts should check that a registration is specified precisely enough for independent implementations to interoperate, and that names are lowercase ASCII and do not duplicate existing entries. 11.2.1. KVSP Problem Types Columns: Type Suffix, Title, Recommended HTTP Status Code, Reference. The type URI is "https://iana.org/assignments/kvsp#" followed by the suffix. Initial contents: the entries of Table 2, with reference RFC XXXX. 11.2.2. KVSP Codecs Columns: Value (16 bits), Name (lowercase ASCII letters, digits, and "-", at most 32 characters), Description, Reference. Initial contents: 0x0000 Reserved; 0x0001 "raw", uncompressed elements in layout "kvsp-layout-1" (Section 4.3). Values 0xFF00-0xFFFF are reserved for Private Use. 11.2.3. KVSP Element Types and Positional Encodings "KVSP Element Types" has columns Name, Description, Reference; initial contents: "bf16" (bfloat16), "fp16" (IEEE 754 binary16), "fp32" (IEEE 754 binary32), "fp8-e4m3", and "fp8-e5m2" (8-bit floating point with 4- or 5-bit exponent). "KVSP Positional Encodings" has columns Name, Parameter Syntax, Description, Reference; initial contents: "rope" (rotary position embedding; optional parameters: "theta=" followed by the base as a decimal integer, and ",scaling=" followed by a scaling method name) and "none". 11.2.4. KVSP Storage Tiers Columns: Name (lowercase ASCII), Description, Reference. Initial contents: "device" (accelerator memory), "host" (host memory, including pinned memory), "local-storage" (storage attached to the holder), "remote-memory" (disaggregated memory pool), "remote- storage" (networked storage). Li, et al. Expires 6 May 2027 [Page 22] Internet-Draft KV Cache Storage Protocol November 2026 11.2.5. KVSP Instance States Columns: Name (lowercase ASCII), Description, Reference. Initial contents: "active", "draining", "sleeping", "failed", as defined in Section 6.3. 12. References 12.1. Normative References [RFC2104] Krawczyk, H., Bellare, M., and R. Canetti, "HMAC: Keyed- Hashing for Message Authentication", RFC 2104, DOI 10.17487/RFC2104, February 1997, . [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC4648] Josefsson, S., "The Base16, Base32, and Base64 Data Encodings", RFC 4648, DOI 10.17487/RFC4648, October 2006, . [RFC6234] Eastlake 3rd, D. and T. Hansen, "US Secure Hash Algorithms (SHA and SHA-based HMAC and HKDF)", RFC 6234, DOI 10.17487/RFC6234, May 2011, . [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type Specifications and Registration Procedures", BCP 13, RFC 6838, DOI 10.17487/RFC6838, January 2013, . [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 8126, DOI 10.17487/RFC8126, June 2017, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data Interchange Format", STD 90, RFC 8259, DOI 10.17487/RFC8259, December 2017, . Li, et al. Expires 6 May 2027 [Page 23] Internet-Draft KV Cache Storage Protocol November 2026 [RFC8446] Rescorla, E., "The Transport Layer Security (TLS) Protocol Version 1.3", RFC 8446, DOI 10.17487/RFC8446, August 2018, . [RFC8785] Rundgren, A., Jordan, B., and S. Erdtman, "JSON Canonicalization Scheme (JCS)", RFC 8785, DOI 10.17487/RFC8785, June 2020, . [RFC9110] Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke, Ed., "HTTP Semantics", STD 97, RFC 9110, DOI 10.17487/RFC9110, June 2022, . [RFC9113] Thomson, M., Ed. and C. Benfield, Ed., "HTTP/2", RFC 9113, DOI 10.17487/RFC9113, June 2022, . [RFC9114] Bishop, M., Ed., "HTTP/3", RFC 9114, DOI 10.17487/RFC9114, June 2022, . [RFC9205] Nottingham, M., "Building Protocols with HTTP", BCP 56, RFC 9205, DOI 10.17487/RFC9205, June 2022, . [RFC9457] Nottingham, M., Wilde, E., and S. Dalal, "Problem Details for HTTP APIs", RFC 9457, DOI 10.17487/RFC9457, July 2023, . [RFC9530] Polli, R. and L. Pardue, "Digest Fields", RFC 9530, DOI 10.17487/RFC9530, February 2024, . [RFC9562] Davis, K., Peabody, B., and P. Leach, "Universally Unique IDentifiers (UUIDs)", RFC 9562, DOI 10.17487/RFC9562, May 2024, . 12.2. Informative References [I-D.ietf-cats-framework] Li, C., Du, Z., Boucadair, M., Contreras, L. M., and J. Drake, "A Framework for Computing-Aware Traffic Steering (CATS)", Work in Progress, Internet-Draft, draft-ietf- cats-framework-24, 2 April 2026, . Li, et al. Expires 6 May 2027 [Page 24] Internet-Draft KV Cache Storage Protocol November 2026 [I-D.li-cats-kv-cache-distribution] Li, Z., Du, Z., Wang, J., Cheng, W., Zhang, G., Sun, X., and C. Zhao, "KV Cache Distribution for Distributed LLM Inference: Use Case and Requirements", Work in Progress, Internet-Draft, draft-li-cats-kv-cache-distribution-00, 4 July 2026, . [I-D.li-kvcache-efficient-transmission] Li, Z., Du, Z., Wang, J., Cheng, W., Zhao, Y., Zhao, C., and X. Sun, "Layer-Aligned Key-Value (KV) Cache Streaming over Media over QUIC Transport (MOQT)", Work in Progress, Internet-Draft, draft-li-kvcache-efficient-transmission- 00, 2 November 2026, . [I-D.li-rdma-kvcache-transfer-optimization] Li, Z., Du, Z., Wang, J., Cheng, W., Zhao, Y., Zhao, C., and X. Sun, "Key-Value (KV) Cache Block Transfer Protocol (KBTP) over RDMA", Work in Progress, Internet-Draft, draft-li-rdma-kvcache-transfer-optimization-00, 2 November 2026, . [MOONCAKE] Qin, R., Li, Z., He, W., Cui, J., Ren, F., Zhang, M., Wu, Y., Zheng, W., and X. Xu, "Mooncake: Trading More Storage for Less Computation - A KVCache-centric Architecture for Serving LLM Chatbot", 23rd USENIX Conference on File and Storage Technologies (FAST 25), February 2025, . [PAGEDATTENTION] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and I. Stoica, "Efficient Memory Management for Large Language Model Serving with PagedAttention", Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP), October 2023, . [RFC6920] Farrell, S., Kutscher, D., Dannewitz, C., Ohlman, B., Keranen, A., and P. Hallam-Baker, "Naming Things with Hashes", RFC 6920, DOI 10.17487/RFC6920, April 2013, . Li, et al. Expires 6 May 2027 [Page 25] Internet-Draft KV Cache Storage Protocol November 2026 [RFC6973] Cooper, A., Tschofenig, H., Aboba, B., Peterson, J., Morris, J., Hansen, M., and R. Smith, "Privacy Considerations for Internet Protocols", RFC 6973, DOI 10.17487/RFC6973, July 2013, . Appendix A. Test Vectors Compatibility Descriptor after JCS canonicalization (one line, wrapped here for presentation): {"block_tokens":4,"head_dim":128,"kv_dtype":"bf16","kv_heads":8, "layers":32,"layout":"kvsp-layout-1","model":"example-model", "model_digest":"sha-256:254e528c669a2084c9727298cacb83e8854e0a6f f80e2bf18096b9dce0f7ebda","positional":"rope","tokenizer_digest" :"sha-256:9f42a7a29dca3063ad26d7be7fe6a930fa38afb1a703e4f7ff538db 4e133b19c"} compat_id = 5ed5c2f238839655d7c4029e12b7f31c af9e7421f4edb4057feee6dab760ee0f Scope Key = octets 0x00, 0x01, ..., 0x1f. Tokens = 128000, 791, 6460, 374, 264, 1296, 315, 279 (two blocks of four): scope_id = 6d1c6481cd428737da312c089db3832a K[0] = 9ca9d8b60f37f84726a27deb4cdc0d07 5714607a131632c2366e30b99fb1163b K[1] = 653275884ca44f8945fd4c902611c467 394b73656e64c78d8522648cb75aefc1 Container header for codec "raw", layers 0-1, four tokens, with K and V sections of 8192 octets each (4 tokens x 8 heads x 128 x 2 octets): 4b 56 42 31 01 00 00 01 00 00 00 00 00 00 00 02 00 00 00 04 00 00 00 00 00 00 20 00 00 00 00 00 00 00 20 00 00 00 00 00 00 00 20 00 00 00 00 00 00 00 20 00 Acknowledgments The authors thank the participants of the CATS and distributed inference discussions for their feedback on the KV cache distribution use case. Authors' Addresses Li, et al. Expires 6 May 2027 [Page 26] Internet-Draft KV Cache Storage Protocol November 2026 Zhiqiang Li China Mobile Beijing 100053 China Email: lizhiqiangyjy@chinamobile.com Zongpeng Du China Mobile Beijing 100053 China Email: duzongpeng@chinamobile.com Junjie Wang Centec Shanghai 201203 China Email: wangjj@centec.com Wei Cheng Centec Shanghai 201203 China Email: chengw@centec.com Ying Zhao Centec Shanghai 200000 China Email: zhaoying@centec.com Chunhao Zhao SAIA Shanghai 200125 China Email: chunhao.zhao@sh-aia.com Li, et al. Expires 6 May 2027 [Page 27] Internet-Draft KV Cache Storage Protocol November 2026 Xun Sun Inesa Shanghai 200030 China Email: sunxun@inesa.com Li, et al. Expires 6 May 2027 [Page 28]