]> Nomotic, Inc.

chris@nomotic.ai https://nomotic.ai

Applications and Real-Time AI agents real-time communication voice video multi-modal This document specifies the AGTP Communication Protocol (AGTP-COMMUNICATION): the companion specification for real-time multi-modal communication between agents over the Agent Transfer Protocol (AGTP). AGTP-COMMUNICATION defines how voice, video, and other real-time media streams are exchanged between agents on the agent-native substrate, with native support for the wire-level identity, authority scope, and attribution that AGTP provides. This is an early specification covering bilateral (two-agent) real-time communication. Multi-party conversations and conferencing patterns are out of scope for this revision and are deferred to future companion work.

Introduction The Agent Transfer Protocol (AGTP) is a transfer protocol for agent-to-agent and agent-to-API communication, the dedicated substrate on which agent traffic runs. AGTP carries agent identity, authority scope, attribution records, and intent-aligned methods at the wire level, with traffic identified as agent traffic by the protocol itself. Agent communication is increasingly multi-modal. Agents communicate through voice when speaking to humans or to other voice-capable agents. Agents communicate through video when participating in visual interactions, screen sharing, or visual data exchange. Agents communicate through structured data streams for sensor data, telemetry, and continuous information flows. These real-time communication patterns require protocol-level support distinct from the request/response patterns AGTP's base methods address. This document specifies how real-time multi-modal communication runs on AGTP. The design reuses established real-time media patterns where appropriate (drawing on the architectural principles of and ) and defines only what is specific to agent-native communication on the AGTP substrate.

Relationship to AGTP-SESSION AGTP-SESSION defines session establishment, lifecycle, resumption, and termination on AGTP. AGTP-COMMUNICATION builds on AGTP-SESSION: a Communication Session is an AGTP session established through ESTABLISH, with media parameters carried as an extension of the ESTABLISH body defined there. AGTP-SESSION's distinction between a connection (what a binding of opens) and a session (what ESTABLISH creates) matters here more than anywhere else in the family: media streams are bound to a connection and end with it, while the Communication Session survives connection loss through RESUME and re-opens its streams.

Scope of This Document In scope:

• Bilateral real-time audio communication between agents
• Bilateral real-time video communication between agents
• Multi-modal exchange (audio plus video, structured data alongside media)
• Codec negotiation and media format selection
• Real-time media framing on the AGTP substrate
• Quality of service handling at the AGTP layer
• Integration with AGTP-SESSION for session lifecycle

Out of scope for this revision:

• Multi-party conversations (three or more agents)
• Conferencing patterns (mixers, SFUs, broadcast)
• Recording and replay protocols
• Voice-specific applications (telephony, IVR patterns)
• Domain-specific conversational AI patterns

Conventions and Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Terminology

Communication Session:

An AGTP-SESSION established for real-time multi-modal communication between two agents, with media parameters negotiated during session establishment.

Media Stream:

A unidirectional flow of real-time media data within a Communication Session. A bilateral Communication Session typically carries two media streams (one in each direction) per modality.

Modality:

A category of real-time media. This specification addresses audio, video, and structured data modalities. Future revisions may address additional modalities.

Codec:

An encoding format for media data, negotiated between communicating agents during session establishment.

Communication Endpoint:

An AGTP-aware agent participating in a Communication Session. Identified by its canonical Agent-ID and carrying authority scope appropriate to the communication being undertaken.

Architectural Model AGTP-COMMUNICATION extends AGTP's request/response model with real-time streaming semantics. The architectural model has three components:

Session Layer Communication Sessions are established using AGTP-SESSION's ESTABLISH method with communication-specific parameters. The session carries the agent identity, authority scope, and attribution chain that apply throughout the communication. Session establishment for communication is more involved than session establishment for request/response: media parameters must be negotiated, codecs agreed, and stream characteristics established before media can flow.

Media Layer Media streams carry real-time data between Communication Endpoints. Each stream has a defined modality (audio, video, or structured data), a negotiated codec, and timing characteristics appropriate to its modality. Media streams are framed for carriage on AGTP. The framing preserves the timing and sequencing properties that real-time media requires while binding each frame to the Communication Session, and through it to the AGTP wire-level facts (identity, authority scope, attribution) established at ESTABLISH. Media frames carry the Session-ID and a stream identifier only, with no AGTP request header set; the per-request proof of possession of is made once, at ESTABLISH and at each control message, and the session token binds the frames that follow to it.

Control Layer Control messages within a Communication Session manage stream lifecycle: opening streams, modifying parameters, handling quality degradation, and closing streams. Control messages use AGTP methods within the established session context.

Communication Session Establishment Communication Sessions are established through the ESTABLISH method of , on the path /session, with the standard AGTP request headers including Request-Signature, and with a communication object added to the ESTABLISH body. Version 00 of this document carried the media parameters in request headers (Session-Intent, Communication-Modalities, Audio-Codecs, Video-Codecs); they move into the body so that one ESTABLISH grammar serves the whole family and the media offer is covered by the same signature and the same session token as the session parameters.

ESTABLISH Request A Communication Endpoint initiates a session by issuing ESTABLISH with a communication offer: Authority-Scope: communication:bilateral Request-Signature: Content-Type: application/json Content-Length: ... { "model": "bounded", "session_ttl": 3600, "scopes": ["communication:bilateral"], "resume_window": 60, "initiator_nonce": "", "communication": { "modalities": ["audio", "video"], "audio_codecs": ["opus", "g722"], "video_codecs": ["vp9", "av1"], "data_streams": [ { "label": "telemetry", "reliable": false, "max_rate_hz": 50 } ] } } ]]> The fields outside communication are those of and keep their semantics. Within communication:

modalities:

The modalities the initiator wishes to use, from audio, video, data. REQUIRED, at least one.

audio_codecs, video_codecs:

Codecs the initiator supports for the modality, in order of preference, by registered codec name. REQUIRED for each of audio and video that appears in modalities.

data_streams:

Structured data streams the initiator proposes, each with a label, a reliable flag, and the rate and schema parameters the application needs. REQUIRED when data appears in modalities.

A resume_window of a minute or less is RECOMMENDED for Communication Sessions: media state is rarely worth retaining longer, and a shorter window bounds the exposure discussed in .

ESTABLISH Response The receiving Communication Endpoint responds with the negotiated parameters or rejects the session: Agent-ID: Session-ID: <64 lowercase hex> Content-Type: application/json { "session_id": "<64 lowercase hex>", "model": "bounded", "expires_at": "2026-10-29T18:00:00Z", "resume_window": 60, "scopes": ["communication:bilateral"], "session_token": "", "communication": { "modalities": ["audio", "video"], "audio_codec": "opus", "video_codec": "vp9", "data_streams": [ { "label": "telemetry", "reliable": false, "max_rate_hz": 50 } ], "stream_parameters": { "...": "..." } } } ]]> Successful establishment returns 200 with the Session-ID header and the negotiated parameters, one codec per modality. The ESTABLISH invocation is the first Attribution-Record under the session and the first entry in its session root per . Rejection uses the status codes of : 455 Scope Violation when the initiator's Authority-Scope lacks a permission the offered modalities require, with the reason codes of ; 422 when no offered codec is acceptable for a required modality or a requested modality is unsupported, with a body naming the modality and the codecs the responder would accept so the initiator can re-offer; 503 when the responder is at capacity or its governance platform cannot be reached to record the session. A responder MUST NOT accept a subset of the offered modalities silently; it either negotiates each modality in modalities or rejects with 422 naming the one it cannot serve. A response that omits the communication key entirely means the responder predates this extension (); the initiator MUST TERMINATE the session; sending media into it is prohibited.

Authority Scope Considerations Communication Sessions carry significant authority implications. A session that includes audio capture and transmission grants the initiating agent the ability to capture and transmit audio for the session duration. Authority-Scope MUST include appropriate permissions for each modality:

• communication:audio:capture for capturing audio
• communication:audio:transmit for transmitting audio
• communication:video:capture for capturing video
• communication:video:transmit for transmitting video
• communication:bilateral as a shorthand combining standard bilateral capture and transmission

Receivers MUST validate that the initiator's Authority-Scope includes appropriate permissions for the requested modalities, and that the session-scoped scopes the initiator proposes are a subset of it, per . A delegated initiator carries Delegation-Chain on ESTABLISH and the receiver verifies it per before the first media frame flows.

Media Stream Semantics Media streams within a Communication Session carry real-time data with timing, sequencing, and quality requirements appropriate to their modality.

Audio Streams Audio streams carry audio media between Communication Endpoints. Audio framing follows established real-time audio practice with adaptation for AGTP transport:

• Frames carry timestamp information for synchronization
• Sequence numbers detect loss and reordering
• Frame size is negotiated during session establishment
• Codec-specific parameters (sample rate, channels) are negotiated

AGTP-COMMUNICATION reuses RTP timestamp and sequence semantics where compatible, adapted for carriage on AGTP in place of UDP. This preserves established real-time audio handling while gaining AGTP's wire-level identity and attribution properties.

Video Streams Video streams carry video media between Communication Endpoints. Video framing addresses the additional complexity of variable frame sizes, key frame management, and bandwidth adaptation:

• Frames carry timestamp and sequence information
• Frame type (key/delta) is indicated
• Codec-specific parameters (resolution, frame rate) are negotiated
• Bandwidth adaptation signals are exchanged through control messages

Structured Data Streams Structured data streams carry continuous data flows other than audio or video: sensor telemetry, conversational state updates, real-time analytics, contextual data alongside other media. Structured data streams have different real-time characteristics than audio or video. Timing may matter (sensor sampling rates) or matter little (state updates). Loss tolerance varies by use case. Structured data stream parameters are negotiated during session establishment through data_streams.

Carriage on the Bindings Which transport features a media stream can use depends on the binding of beneath the connection. On the QUIC binding, each media stream SHOULD be carried on its own QUIC unidirectional stream when the stream is marked reliable, and on QUIC DATAGRAM frames when it is marked unreliable, so that loss in one stream never blocks another. On the TCP binding, every stream shares one ordered byte stream; head-of-line blocking applies and the latency figures of are unlikely to be met, so the TCP binding is suitable for structured data and for audio where latency tolerance is known to be wide. On the HTTP/3 and WebTransport binding, WebTransport unidirectional streams and datagrams serve the same roles as their QUIC counterparts. Media streams belong to the connection. When the connection is lost and the Communication Session is resumed through RESUME per , every media stream is closed and the endpoints re-open the streams they still need through control messages; the negotiated codecs and parameters carry over with the session and need no renegotiation.

Quality of Service Real-time communication has quality requirements that the binding beneath AGTP must meet. AGTP-COMMUNICATION specifies quality of service handling appropriate to each modality.

Latency Requirements Audio communication typically requires latency under 150ms for natural conversational flow. Video communication tolerates higher latency but synchronization between audio and video is critical. Structured data streams have application-specific latency requirements. When AGTP runs over QUIC , the underlying transport supports multiple streams with independent flow control, which enables appropriate handling of different modality requirements within a single Communication Session.

Bandwidth Adaptation Communication Endpoints MUST be capable of adapting media parameters in response to bandwidth constraints. Control messages within a Communication Session signal:

• Bandwidth estimates from the receiving endpoint
• Requested adaptations from the sending endpoint
• Confirmation of parameter changes

Bandwidth adaptation is negotiated; both endpoints participate in the decision to adapt.

Priority Within AGTP AGTP traffic on port 4480 SHOULD be treated by the network with priority appropriate to its modality. Real-time audio and video streams require lower latency than request/response traffic; structured data streams may have varying requirements. Network operators carrying AGTP traffic SHOULD consider that AGTP-COMMUNICATION sessions are likely to include latency-sensitive real-time media and apply appropriate QoS handling.

Attribution and Recording AGTP's attribution model applies to Communication Sessions: the ESTABLISH invocation, every control message (stream open, parameter change, stream close), RESUME, and TERMINATE each produce an Attribution-Record, and the session root of summarizes them. Media frames produce no Attribution-Records of their own; the stream-open record names the stream, its modality and codec, and the Agent-IDs on each end, and the stream-close record carries the frame and byte counts, so the existence, extent, and parties of every media exchange are attributable without logging the media. Media content within streams is unrecorded by the protocol by default. Recording is an application-layer decision made by governance frameworks or specific deployments. AGTP-COMMUNICATION provides the session-level attribution that recording systems can build on; it performs no recording itself. When recording is performed at the application layer, the attribution records produced by AGTP-COMMUNICATION provide verifiable evidence of session participants, authority scope, and session lifecycle that supports compliance with recording-relevant regulations.

Security Considerations Real-time communication on AGTP inherits AGTP's security properties: encryption from the binding (TLS 1.3 or QUIC), agent identity verification including per-request proof of possession, and authority scope enforcement at the protocol layer. Additional security considerations specific to communication:

Media Capture Authorization Agents that capture audio or video MUST have appropriate Authority-Scope. This is enforced at session establishment. Capture without scope is a 455 Scope Violation.

Replay and Tampering Audio and video streams MUST NOT be replayable across sessions without the cryptographic markers that identify them as recordings. Session identifiers, timestamps, and attribution records carried with streams enable verification that media was captured in the context the recipient believes.

Privacy Considerations Communication Sessions may involve sensitive content (private conversations, confidential video, sensor data with privacy implications). AGTP's wire-level identity verification and attribution provide the structural facts that privacy frameworks require. Application-layer privacy controls build on these foundations.

Denial of Service Real-time communication can be used to consume substantial bandwidth and processing resources. Communication Endpoints SHOULD implement appropriate rate limits and resource controls. Authority-Scope can include resource limitations that the protocol enforces at session establishment.

IANA Considerations This document defines no new headers. It requests, in the registries established by and once they exist:

• The communication key in the ESTABLISH body extension registry of , with this document as reference
• Authority-Scope tokens for communication (communication:audio:capture, communication:audio:transmit, communication:video:capture, communication:video:transmit, communication:bilateral)

Codec names in audio_codecs and video_codecs are drawn from the RTP payload format registries ; this document creates no codec registry.

Open Questions Several design decisions remain open for this revision:

• Whether to define an AGTP-specific real-time media framing or to reuse RTP framing carried over AGTP transport
• The relationship to WebRTC for browser-based agents communicating over AGTP
• Whether to define agent-specific codecs (e.g., for low-bandwidth agent-to-agent voice with no need to sound human) or to rely entirely on existing codec registries
• How AGTP-COMMUNICATION sessions interact with AGTP's intent methods for non-real-time exchanges within the same agent pair
• Multi-party conversation patterns and whether they belong as a v01 extension or as a separate companion specification

These will be addressed in future revisions of this draft based on community feedback and implementation experience.

Changes from v00 Version 01 is an alignment revision; the architectural model, modalities, scope tokens, and open questions are unchanged.

• Informational. Version 00 was marked Standards Track; the document is now Informational, matching every other draft in the AGTP family.
• ESTABLISH aligned to AGTP-SESSION (, ). The media offer moves from request headers into a communication object in the ESTABLISH body defined by draft-hood-agtp-session-01; the request line, path, Request-Signature, Session-ID, and session token are those of that document. The four v00 headers are withdrawn.
• Status codes. 451 becomes 455 Scope Violation, matching the renumbering in draft-hood-independent-agtp-10; parameter mismatch is 422 with a re-offer body; 463, which v00 used, is the code base AGTP reserves for PROPOSE.
• Carriage on the bindings (): QUIC streams and datagrams per modality, the TCP binding's limits, and the rule that media streams belong to the connection and are re-opened after RESUME.
• Attribution restated around the session root: control messages are attributed, media frames are summarized by the stream-close record.
• References pinned to draft-hood-independent-agtp-10 and draft-hood-agtp-session-01; AGTP-BINDINGS and RFC 9221 added. Transfer-protocol wording applied. House style pass.

References Normative References In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. This memorandum describes RTP, the real-time transport protocol. RTP provides end-to-end network transport functions suitable for applications transmitting real-time data, such as audio, video or simulation data, over multicast or unicast network services. RTP does not address resource reservation and does not guarantee quality-of- service for real-time services. The data transport is augmented by a control protocol (RTCP) to allow monitoring of the data delivery in a manner scalable to large multicast networks, and to provide minimal control and identification functionality. RTP and RTCP are designed to be independent of the underlying transport and network layers. The protocol supports the use of RTP-level translators and mixers. Most of the text in this memorandum is identical to RFC 1889 which it obsoletes. There are no changes in the packet formats on the wire, only changes to the rules and algorithms governing how the protocol is used. The biggest change is an enhancement to the scalable timer algorithm for calculating when to send RTCP packets in order to minimize transmission in excess of the intended rate when many participants join a session simultaneously. [STANDARDS-TRACK] RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This document gives an overview and context of a protocol suite intended for use with real-time applications that can be deployed in browsers -- "real-time communication on the Web". It intends to serve as a starting and coordination point to make sure that (1) all the parts that are needed to achieve this goal are findable and (2) the parts that belong in the Internet protocol suite are fully specified and on the right publication track. This document is an applicability statement -- it does not itself specify any protocol, but it specifies which other specifications implementations are supposed to follow to be compliant with Web Real-Time Communication (WebRTC). Informative References This document describes a profile called "RTP/AVP" for the use of the real-time transport protocol (RTP), version 2, and the associated control protocol, RTCP, within audio and video multiparticipant conferences with minimal control. It provides interpretations of generic fields within the RTP specification suitable for audio and video conferences. In particular, this document defines a set of default mappings from payload type numbers to encodings. This document also describes how audio and video data may be carried within RTP. It defines a set of standard encodings and their names when used within RTP. The descriptions provide pointers to reference implementations and the detailed standards. This document is meant as an aid for implementors of audio, video and other real-time multimedia applications. This memorandum obsoletes RFC 1890. It is mostly backwards-compatible except for functions removed because two interoperable implementations were not found. The additions to RFC 1890 codify existing practice in the use of payload formats under this profile and include new payload formats defined since RFC 1890 was published. [STANDARDS-TRACK] The terminology about, and associations among, Real-time Transport Protocol (RTP) sources can be complex and somewhat opaque. This document describes a number of existing and proposed properties and relationships among RTP sources and defines common terminology for discussing protocol entities and their relationships. This document defines the core of the QUIC transport protocol. QUIC provides applications with flow-controlled streams for structured communication, low-latency connection establishment, and network path migration. QUIC includes security measures that ensure confidentiality, integrity, and availability in a range of deployment circumstances. Accompanying documents describe the integration of TLS for key negotiation, loss detection, and an exemplary congestion control algorithm. This document defines an extension to the QUIC transport protocol to add support for sending and receiving unreliable datagrams over a QUIC connection.

Acknowledgments This document builds on the broader AGTP family and incorporates architectural principles from established real-time media work including RTP/RTCP and WebRTC .

Contributors Contributors will be acknowledged in future revisions as community participation develops.