China Mobile

Beijing100053 China lizhiqiangyjy@chinamobile.com

China Mobile

Beijing100053 China duzongpeng@chinamobile.com

Centec

Shanghai201203 China wangjj@centec.com

Centec

Shanghai201203 China chengw@centec.com

Centec

Shanghai200000 China zhaoying@centec.com

SAIA

Shanghai200125 China chunhao.zhao@sh-aia.com

Inesa

Shanghai200030 China sunxun@inesa.com

Web and Internet Transport MOQT Media over QUIC KV cache relay caching publish/subscribe distributed inference The key-value (KV) cache of a language model can be computed once and reused by many inference instances, possibly at many sites. Media over QUIC Transport (MOQT) provides publish/subscribe delivery through relays that cache objects without understanding their content. This document specifies KV Cache Streaming (KVCS), an application of MOQT for delivering KV caches. KVCS maps the identity of a token prefix to a Track Namespace built from keyed identifiers that do not reveal the prompt, carries the key and value tensors (or a jointly encoded form) in separate Tracks, aligns each Group with one model layer and each Object with one block of tokens, and defines a self-describing object payload and a manifest track. These choices let consumers start using early layers while later layers are still being produced, retrieve only the layers and token blocks they need, and verify what they receive.

Introduction

Motivation Inference systems increasingly treat the KV cache as state that is decoupled from the engine that computed it: a prefill instance or cache server produces it, and decode instances, other replicas, or other sites consume it. The same prefix (for example a long system prompt or a shared document) is often needed by many consumers. Point-to-point transfer from the producer to each consumer wastes backbone capacity and loads the producer. MOQT already solves the analogous problem for media: an original publisher sends objects once to a relay tree, relays cache them, and many subscribers receive them with per-object priorities, without the relays understanding the payload. A KV cache has a natural hierarchy that fits the MOQT object model: a cache belongs to a prefix, consists of key and value tensors, is computed layer by layer, and is divided into blocks of tokens.

Summary of the Profile

Mapping of a KV Cache to the MOQT Object Model Track Namespace ("kvcs-1", scope_id, compat_id, anchor_key, pub_id) | +-- Track "manifest" Group m: manifest version m | +-- Track "k.raw.s0" key tensors, codec "raw", segment 0 | +-- Group 0 layer 0 | | +-- Object 0 tokens of block 0, layer 0 | | +-- Object 1 tokens of block 1, layer 0 | | +-- ... | +-- Group 1 layer 1 | +-- ... | +-- Group L-1 layer L-1 | +-- Track "v.raw.s0" value tensors, same structure +-- Track "kv.<codec>.s0" jointly encoded K and V (optional) +-- Track "k.raw.s1" ... segment 1 (appended tokens)

The design choices are:

• Namespace from identity, not from text. The namespace carries the keyed identifiers of the KV Cache Storage Protocol (KVSP) , so relays and observers learn neither the model name nor the prompt, and two different prompts share a namespace only with negligible probability ().
• Component tracks. Key, value, and jointly encoded tensors are separate tracks, so a consumer can subscribe to what it needs and a publisher can offer several encodings ().
• Layer-aligned groups. A Group is one layer. Production proceeds layer by layer, so Group IDs increase over time as MOQT expects, and a consumer can compute layer l while layer l+1 is still arriving ().
• Block objects. An Object is the tensor of one token block in one layer, the smallest unit that is cached by relays and that can be selected with object filters ().
• Self-describing payloads. Every Object carries its own descriptor and digest, so a subscriber that joins mid-track can use every Object it receives ().

Relationship to Other Work also describes KV cache transmission over MOQT. The present document differs in several respects that affect interoperability and security: (1) it does not place the model name or prompt in the namespace but keyed identifiers, which avoids disclosure and keeps different prompts in different tracks; (2) it aligns Groups with layers rather than with token chunks, so that consumers can pipeline computation across layers and Group IDs increase with production time; (3) it separates key, value, and jointly encoded tensors into tracks and identifies codecs through a registry instead of fixed compression levels; and (4) it defines the object payload, a manifest, integrity checks, and retrieval procedures for both live and completed caches. KVCS depends on KVSP for the Compatibility ID, Scope ID, Block Keys, and the codec registry. Object payload protection uses when relays are not trusted with KV cache contents.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document uses the MOQT terms Original Publisher, Relay, End Subscriber, Track, Track Namespace, Track Name, Group, Subgroup, Object, Location, Largest Object, Object Status, and Track Property as defined in , and the KVSP terms Compatibility ID, Scope ID, Block Key, Block, and codec as defined in . In addition:

Anchor Key:

The Block Key of the last complete block of segment 0, which identifies the token prefix of the publication.

Publication:

One production of the KV cache of a prefix by one Original Publisher, identified by a Publication ID.

Segment:

A contiguous range of complete blocks of the context. Segment 0 covers the prefix identified by the Anchor Key; later segments cover appended tokens.

Component:

The tensor kind carried by a track: "k" (keys), "v" (values), or "kv" (keys and values jointly encoded).

Object Model Mapping

Track Namespace A KVCS Track Namespace consists of exactly five Track Namespace Fields: KVCS Track Namespace Fields

Field	Length	Content
0	6	ASCII "kvcs-1"
1	16	Scope ID
2	32	Compatibility ID
3	32	Anchor Key
4	16	Publication ID

Fields 1 to 4 are raw octets, not their hexadecimal encodings. The Publication ID is chosen at random by the Original Publisher for each production of the cache. Fields 0 to 3 identify content: any consumer that knows the token prefix, the scope key, and the Compatibility Descriptor can compute them. Field 4 is needed because MOQT requires that an Object have identical bytes regardless of where it is retrieved (), while two independent computations of the same prefix are not guaranteed to be bit-identical on all hardware. A consumer that knows fields 0 to 3 discovers the available publications through namespace prefix subscription (). Namespace prefix matching gives useful aggregation points: the prefix ("kvcs-1", Scope ID) covers all publications of a scope and is the natural unit of authorization; the prefix ("kvcs-1", Scope ID, Compatibility ID, Anchor Key) covers all publications of one token prefix.

Track Names Track Names are ASCII strings with the following syntax, in ABNF : track-name = "manifest" / "manifest.cose" / data-track data-track = component "." codec-name "." "s" segment component = "k" / "v" / "kv" codec-name = 1*32( %x61-7A / DIGIT / "-" ) ; KVSP codec name segment = "0" / ( %x31-39 *DIGIT ) ; no leading zeros codec-name is the Name of a codec in the "KVSP Codecs" registry . For example, "k.raw.s0" carries the uncompressed key tensors of segment 0. A publisher MAY offer the same segment in several encodings, for example "k.raw.s0" and "v.raw.s0" together with a "kv" track using a registered compressing codec; each data track is complete on its own. A consumer selects either the pair of "k" and "v" tracks of one codec or one "kv" track.

Groups The Group ID of a data track is the layer index, from 0 to L-1, where L is the number of layers in the Compatibility Descriptor. The Original Publisher MUST publish the Objects of Group g before those of Group g+1 in the same track. Producers that compute the cache in several passes over the token sequence (for example chunked prefill, where each chunk passes through all layers) MUST publish each pass as a separate segment, so that Group IDs within every track increase with time as recommended by . The Objects of a Group are independently usable: each carries the tensor of one block for that layer. A Group is a natural join point because a consumer needs every block of a layer before computing attention for that layer. The Original Publisher sends all Objects of a Group in a single Subgroup with Subgroup ID 0, in ascending Object ID order. After the last Object of Group g it sends an Object with status End of Group. After the last Group of a track it sends an Object with status End of Track.

Objects The Object ID is the index of the block within the segment, starting at 0. The absolute block index is the first block index of the segment, given in the manifest, plus the Object ID. Each Object carries one KVCS payload (). Objects are never modified; a publisher that discovers an error in a publication stops it and starts a new publication with a new Publication ID. The Original Publisher SHOULD set the Publisher Priority of the Objects in Group g to floor(128 * g / L), so that, under contention, earlier layers, which a consumer needs first, are sent first. The manifest track uses Publisher Priority 0.

Object Payload

KVCS Object Payload 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Magic ("KVO1" = 0x4B564F31) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Version | Component | Flags | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Layer | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Block Index | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Token Start (64 bits) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Token Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Codec | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Digest (SHA-256, 32 octets) | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data (variable) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Magic:

0x4B564F31.

Version:

1.

Component:

1 for "k", 2 for "v", 3 for "kv".

Flags:

Reserved; set to zero and ignored on receipt.

Layer:

The layer index; equal to the Group ID.

Block Index:

The index of the block within the segment; equal to the Object ID.

Token Start:

The absolute position in the context of the first token of the block.

Token Count:

The number of tokens; equal to the block size of the Compatibility Descriptor.

Codec:

A value from the KVSP Codecs registry.

Data Length:

The length of Data in octets.

Data Digest:

SHA-256 of Data.

Data:

The tensor of the block for this layer. For codec "raw" and component "k" or "v", this is the K or V section of the KVSP container for one layer; for component "kv" the K section is followed by the V section.

For codec "raw", the SHA-256 of the Data of a "k" Object followed by the Data of the "v" Object at the same Location, or the Data Digest of the corresponding "kv" Object, equals the KVSP layer digest of that block and layer, so that the two documents protect the same content. The header is 68 octets. An End Subscriber MUST verify that Magic and Version are as specified, that Layer equals the Group ID, that Block Index equals the Object ID, that the header and Data Length account for the whole payload, and that the Data Digest matches. An Object that fails any check MUST be discarded and treated as missing.

Manifest Track The manifest track describes the publication. Each version of the manifest is Object 0 of a new Group, with Group IDs increasing by one per version. A publication has exactly one manifest track: "manifest", whose payload is a JSON object, or "manifest.cose", whose payload is a COSE_Sign1 structure with that JSON object as its payload. The JSON object is: { "kvcs": 1, "compat": { ...KVSP Compatibility Descriptor... }, "layers": 32, "block_tokens": 16, "deterministic": false, "segments": [ { "segment": 0, "first_block": 0, "blocks": 64, "block_keys_sha256": "<hex32>", "tracks": ["k.raw.s0", "v.raw.s0"], "complete": true }, { "segment": 1, "first_block": 64, "blocks": 3, "block_keys_sha256": "<hex32>", "tracks": ["k.raw.s1", "v.raw.s1"], "complete": false } ] } "block_keys_sha256" is the SHA-256 of the concatenated raw Block Keys of the segment, which lets a consumer that has computed the keys confirm that the segment covers its tokens. A segment entry SHOULD also carry "object_digests": for each data track, an array with one element per completed Group, in Group order, each element being the array of the hex Data Digests of the Group's Objects in Object order. During production, the publisher SHOULD publish a new manifest version at least every eight completed layers, so that a consumer can verify layers while later ones are still produced. Because every Object has its own listed digest, consumers can also verify partial retrievals (a range of layers or of blocks). With "manifest.cose", the publisher signs each version with its producer key, using the COSE algorithm ES256 or EdDSA , and places the key identifier in the "kid" header parameter. Producer keys are distributed to the members of the scope together with the scope key, by means outside this document. "deterministic" is true only if the publisher guarantees that any publication of the same content fields produces bit-identical Objects. A new manifest version is published whenever a segment is added or becomes complete. The manifest MUST be consistent with the Compatibility ID in the namespace: a consumer MUST compute the Compatibility ID of "compat" and discard the publication if it differs.

Publisher Behavior

• The Original Publisher MUST announce the full five-field namespace of each publication with PUBLISH_NAMESPACE, carrying the authorization the relay requires, so that relays can answer namespace subscriptions for any of its prefixes.
• It publishes the manifest before or together with the first data Object.
• It MUST NOT set OBJECT_DELIVERY_TIMEOUT or SUBGROUP_DELIVERY_TIMEOUT on data tracks: a KV cache with missing objects cannot be used, so timeouts only waste capacity. It SHOULD set the Track Property MAX_CACHE_DURATION to the retention allowed for the scope.
• During live production it publishes each Object as soon as the layer of the block is computed, in the order of .
• It MUST NOT publish two different payloads at the same Location. Combined with the Publication ID, this keeps every Object immutable as MOQT requires.

Consumer Procedures

Discovering Publications A consumer computes the Compatibility ID, Scope ID, and Block Keys of its prompt with KVSP. For the longest prefix it wants to reuse, it uses the Block Key of that prefix's last block as the Anchor Key and sends SUBSCRIBE_NAMESPACE for the prefix ("kvcs-1", Scope ID, Compatibility ID, Anchor Key). Each NAMESPACE response identifies one publication. Alternatively, a KVSP Index Service lookup can return replica records whose holder URI is an MOQT URI identifying the full namespace. If no publication exists for the full prefix, the consumer can try shorter prefixes, because the Anchor Key of a shorter prefix is an earlier Block Key in the same chain. A publication whose prefix is longer than the consumer's needs cannot be found this way, because the consumer cannot compute its Anchor Key; the consumer finds such publications through a KVSP Index Service lookup of its own Block Keys, whose replica records name the publication.

Retrieval during Production To receive a cache that is still being produced, and to receive every Object exactly once, the consumer subscribes to each needed data track with:

• a LOCATION_FILTER of type Next Object (0x05); and
• a FILL_PARAMETERS parameter containing a LOCATION_FILTER of type Absolute Start (0x02) with StartGroup 0 and StartObject 0, which retrieves everything published before the subscription on a fill fetch stream.

This combination is the exactly-once pattern described in . The consumer also subscribes to the manifest track: first to "manifest.cose", and, only if the publisher reports that this track does not exist and the consumer does not require producer authenticity, to "manifest". It uses a LOCATION_FILTER of type Next Object (0x05) and a FILL_PARAMETERS parameter containing a LOCATION_FILTER of type Relative Start (0x01) with StartGroup 1, which delivers the current manifest on the fill fetch stream and later versions on the subscription. A data track is complete for the consumer when it has received End of Track, or PUBLISH_DONE indicating that the track ended, or when the fill fetch stream has ended with a FIN and every Group listed in a manifest that marks the segment complete has been received. Object Status values are not carried on fetch streams, so the last condition is the one that applies to a consumer that subscribes after production has finished.

Retrieval of Completed Caches To retrieve layers a through b of a completed segment, the consumer sends FETCH for each needed data track with a LOCATION_FILTER of type Absolute Start, Group End (0x03) with StartGroup a, StartObject 0, and EndGroupDelta b-a, and with GROUP_ORDER Ascending. The publisher or relay delivers Groups in ascending order, so the consumer can start computing layer a while later layers arrive. A consumer that needs only the first n blocks of a segment, because its prompt diverges after them, adds an OBJECTID_FILTER with the range 0 to n-1 when the peer's MAX_FILTER_RANGES setup option is non-zero; otherwise it discards the excess Objects.

Verification and Recovery The consumer verifies every Object as described in and every manifest as described in . A layer is usable when all Objects that the consumer needs in the corresponding Group, in all selected tracks, have been received and verified. The consumer MUST NOT combine Objects of different publications in one segment unless the manifests of both publications state "deterministic": true. If Objects are missing (for example, reported as an unknown range in a fetch stream, or the stream is reset), the consumer retries with another relay or publication, retrieves the blocks with KVSP over HTTP, or recomputes the affected tokens. It does not wait indefinitely: it SHOULD bound recovery by the time that recomputation would take.

Relay Considerations KVCS requires no relay functions beyond those of MOQT. Relays cache KVCS Objects by Full Track Name and Location like any other Objects and fill gaps with upstream fetches. Because each Object is immutable and the namespace binds it to a single publication, a relay can serve the same Objects to every authorized subscriber of the prefix. Relays MUST enforce authorization at least at the granularity of the namespace prefix ("kvcs-1", Scope ID), so that subscribers cannot obtain caches of a scope they are not authorized for. Relays honor MAX_CACHE_DURATION. shows an example in which a consumer attached to Relay B receives a cache that has partly been cached at Relay B and is partly still being produced.

Example Delivery through Two Relays Producer Relay A Relay B Consumer | PUBLISH_NAMESPACE | | | | (5-field ns) | | | |-------------------->|--------------->| | | | | SUBSCRIBE | | | | "k.raw.s0", | | | | Next Object + | | | | fill from 0/0 | | | |<----------------| | | SUBSCRIBE | | | |<---------------| | | SUBSCRIBE | | fill stream: | |<--------------------| | groups 0..4 | | | | from cache | | | |---------------->| | group 5 objects | | | |-------------------->| group 5 | group 5 | | |--------------->|---------------->| | group 6 ... | | | | End of Track |--------------->|---------------->|

Operational Considerations

• Object size. An Object holds Token Count x kv_heads x head_dim elements; with 16 tokens, 8 heads, 128 dimensions, and two octets per element, a "k" Object carries 32 KiB. Block sizes that give Objects between 16 KiB and 1 MiB balance per-object overhead against granularity.
• Streams. Each Group of each subscribed track uses one Subgroup stream, so a consumer of two tracks of an 80-layer model can see 160 streams per segment. Endpoints SHOULD grant enough concurrent unidirectional streams for the number of layers they pipeline.
• Retention. Publishers SHOULD align MAX_CACHE_DURATION with the retention of the corresponding KVSP replicas.
• Fallback. Consumers SHOULD be able to fall back to HTTP retrieval through KVSP when no MOQT path is available.

Security Considerations The security considerations of and of apply. Confidentiality. MOQT relays can read Object payloads. KV caches reveal information about the prompts from which they were computed. When relays are not trusted with the contents of a scope, publishers MUST protect Object payloads end to end with , using keys distributed to the authorized members of the scope. Integrity. The Data Digest detects accidental corruption. A relay that is not trusted could replace both data and digest; protection against such a relay requires the authenticated encryption of secure objects. Consumers MUST NOT rely on the Data Digest alone when relays are not trusted. Secure objects authenticate membership of the scope, not the identity of the producer: any member of the scope could publish a poisoned cache. When producer authenticity matters, publishers MUST use the "manifest.cose" track with "object_digests" (), and consumers MUST verify the signature, and the Data Digest of every Object against the signed list, before using the Object. Authorization. Relays MUST authorize PUBLISH_NAMESPACE, SUBSCRIBE_NAMESPACE, SUBSCRIBE, and FETCH for KVCS namespaces by scope. A publisher that is not authorized for a scope could otherwise publish poisoned caches under the scope's identifiers. Resource use. A consumer can request many layers and tracks. Relays apply the resource protections of MOQT, and publishers SHOULD limit the number of concurrent fetches per subscriber.

Privacy Considerations The namespace contains keyed identifiers that cannot be linked to a prompt without the scope key, and track names reveal only the component, codec, and segment number. The number of Groups and Objects and their sizes still reveal the number of layers and the length of the context. Deployments that consider this sensitive can pad segments to a fixed number of blocks. Anchor Keys are deterministic within a scope, so a relay can tell that two subscribers requested the same prefix, and a sequence of subscriptions for successively shorter prefixes () reveals where their prompts diverge from published ones. Relays learn this only for scopes whose traffic they carry; deployments that do not trust relays with such linkage use KVSP lookups to find exact publications instead of probing prefixes.

IANA Considerations IANA is requested to create a registry named "KVCS Components" within the "KV Cache Storage Protocol (KVSP)" registry group created by . Each entry has a name (lowercase ASCII), a numeric value (8 bits) used in the Component field, a description, and a reference. The registration policy is Specification Required . Initial contents: Initial KVCS Components

Name	Value	Description
(reserved)	0	Reserved
k	1	Key tensors
v	2	Value tensors
kv	3	Key and value tensors jointly encoded

The names "manifest" and "manifest.cose" are reserved as Track Names and are not components. This document makes no request of MOQT registries.

References Normative References Cisco Google Google Meta Cisco Cisco Cisco Informative References Huawei Technologies China Mobile

Encoding Examples Using the values of the KVSP test vectors (Scope ID 6d1c6481cd428737da312c089db3832a, Compatibility ID 5ed5c2f2...ee0f, Anchor Key 65327588...efc1) and Publication ID 0123456789abcdef0123456789abcdef, the Track Namespace is encoded as follows (108 octets: a field count of 5, then each field prefixed by its length): 05 06 6b 76 63 73 2d 31 10 6d 1c 64 81 cd 42 87 37 da 31 2c 08 9d b3 83 2a 20 5e d5 c2 f2 38 83 96 55 d7 c4 02 9e 12 b7 f3 1c af 9e 74 21 f4 ed b4 05 7f ee e6 da b7 60 ee 0f 20 65 32 75 88 4c a4 4f 89 45 fd 4c 90 26 11 c4 67 39 4b 73 65 6e 64 c7 8d 85 22 64 8c b7 5a ef c1 10 01 23 45 67 89 ab cd ef 01 23 45 67 89 ab cd ef The payload header of a "k" Object at Group 0, Object 1 (Token Start 4, Token Count 4, codec "raw") is shown below. For brevity the example uses 16 octets of Data with values 0x00 to 0x0f; with the geometry of the KVSP test vector, real Data would hold 4 x 8 x 128 x 2 = 8192 octets. 4b 56 4f 31 01 01 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 04 00 00 00 04 00 01 00 00 00 00 00 10 be 45 cb 26 05 bf 36 be bd e6 84 84 1a 28 f0 fd 43 c6 98 50 a3 dc e5 fe db a6 99 28 ee 3a 89 91

Acknowledgments The authors thank the MOQ working group participants whose discussions of non-media uses of MOQT informed this document.