The Signed-Agent-Metadata HTTP Header Field for Web Bot Auth

Introduction

Problem Statement: Shared Fate on Multi-Tenant Platforms Web Bot Auth () allows automated HTTP clients to cryptographically sign requests using HTTP Message Signatures () and advertise their key discovery URI in the Signature-Agent header field. This model maps cleanly when a single organization operates a single crawler or bot with robust abuse controls. However, cloud browser providers, hosted AI agent platforms, and enterprise gateways run thousands of independent customer workloads behind shared egress infrastructure. Consider a fictional Acme Corp (platform.acme.com), which has pivoted from mail-order industrial products into hosting cloud AI agents. When Acme signs outbound requests with its platform domain key, all traffic from its customers shares one Signature-Agent identity: This creates a shared-fate problem for both platforms and verifiers (origins, CDNs, and security services). Suppose one tenant runs an aggressive scraping workload against shopping.com, while hundreds of other tenants run benign research and shopping agents on the same platform. The verifier at shopping.com observes malicious traffic authenticated only by platform.acme.com. Without sub-identifiers, the verifier must choose between two poor outcomes:

Penalize the entire platform: Apply rate limits, challenges, or blocks to all requests from platform.acme.com, which disrupts legitimate tenants sharing the platform.
Permit platform-level abuse: Allow traffic from platform.acme.com to bypass controls, which leaves shopping.com exposed to the abusive tenant.

Platforms and verifiers need a lightweight mechanism to disambiguate sub-identities and convey per-request operational metadata under a shared Signature-Agent key without exposing personally identifiable information (PII) on the wire.

Relationship to Web Bot Auth and Signature Agent Cards The Web Bot Auth ecosystem separates cryptographic key discovery, static agent description, and dynamic per-request metadata:

Base Protocol (): Defines how an automated client signs HTTP requests and points verifiers to its public keys via Signature-Agent.
Signature Agent Card (): Defines a hosted JSON metadata document where an agent operator advertises static, operator-level properties, including client_name, contacts, and a web_bot_auth object (purpose, trigger, rate-expectation, rfc9309-product-token, and rfc9309-compliance).
Per-Request Metadata (Signed-Agent-Metadata, this document): Defines an in-band HTTP header field that carries dynamic, pseudonymous sub-identifiers (product, project, session) covered by the per-request HTTP Message Signature.

Signed-Agent-Metadata (SAM) is additive to the Signature Agent Card. Static properties that describe the agent as a whole (such as purpose or trigger) belong in the hosted Signature Agent Card and are intentionally omitted from Signed-Agent-Metadata to avoid redundancy. Similarly, if an operator already publishes a separate Signature-Agent URI and Signature Agent Card for each product, the operator does not need to include the product member in Signed-Agent-Metadata.

Related Approaches and Use Cases Several existing or proposed mechanisms address adjacent bot authentication problems, each targeting a different operational model:

Mechanism	Primary Use Case	Limitation for Multi-Tenant Shared Keys
Distinct `Signature-Agent` URIs and Keys	First-party crawlers, single-tenant bots, or distinct product lines where the operator hosts a dedicated JWK Set per identity.	Provisioning, hosting, and caching separate key directories for thousands of dynamic customer projects or millions of ephemeral sessions creates high operational and key-fetch overhead.
HTTP Cookies	Maintaining stateful session continuity within a single browser instance on a single origin after an initial HTTP response.	Cookies are absent on the first request to an origin, are not persisted by stateless HTTP fetchers, and can be cleared or modified by the client workload. A signed header provides platform-attested continuity from the first request.
Anonymous Rate-Limiting Tokens (e.g., Privacy Pass )	Locally running agents or privacy-sensitive clients proving attestation or rate-limit compliance without linkable identifiers.	Unlinkable tokens do not allow a verifier to isolate and block a specific abusive tenant mid-attack, nor do they give the platform an identifier to investigate when an origin reports abuse.

Signed-Agent-Metadata specifically addresses multi-tenant platforms that sign traffic under a common origin and need low-overhead sub-identity or specific behavior isolation.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document uses the following terms:

Platform: An entity that operates agent execution infrastructure and signs outbound HTTP requests with its Signature-Agent key on behalf of multiple products, customer projects, customers, or sessions/workflows.
Tenant: A customer organization, developer account, or billing workspace that deploys or runs automated workloads on a Platform.
Verifier: An HTTP origin server, reverse proxy, Content Delivery Network (CDN), or security provider that validates the HTTP Message Signature and evaluates the Signed-Agent-Metadata header field.
Effective Identity: The hierarchical tuple formed by the verified Signature-Agent URI and the signed Signed-Agent-Metadata identifier members present on a request, ordered from broadest to narrowest scope: (Signature-Agent, product, project, session).

The Signed-Agent-Metadata Header Field

Syntax The Signed-Agent-Metadata HTTP header field is a Dictionary Structured Field as defined in Section 3.2 of . Its ABNF syntax follows the sf-dictionary production rule: Each dictionary member key is a lowercase token (key in ). Each member value defined in this specification MUST be a String (sf-string in ). Parameters on dictionary members are not defined by this specification and MUST be ignored by verifiers. An example Signed-Agent-Metadata header field with all three members present:

Dictionary Members Sending the Signed-Agent-Metadata header field is OPTIONAL, and all dictionary members within it are OPTIONAL. A platform MAY include any subset of members that fits its architecture and tenancy model to reduce shared-fate penalties across its workloads.

Member Key	Type	Description	Example
`product`	`sf-string`	Identifier or application name of the agent software or service under the signing platform.	`product="Acme-Labs"`
`project`	`sf-string`	Opaque identifier for the tenant account, workspace, or billing project on the host platform.	`project="prj_9012"`
`session`	`sf-string`	Ephemeral, opaque identifier for a single bounded task execution, workflow run, or isolated browser context lifecycle.	`session="run_3a2b1c"`

Verifiers MUST ignore unrecognized member keys in the Signed-Agent-Metadata dictionary to allow backward-compatible extensions.

Hierarchical Scoping and Chaining Identifiers carried in Signed-Agent-Metadata are scoped namespaces, not globally unique identifiers. They form a strict hierarchy under the signing Signature-Agent: When evaluating Signed-Agent-Metadata, verifiers MUST apply the following scoping rules:

Chained Tuple Evaluation: A narrower identifier MUST always be evaluated within the namespace of the verified Signature-Agent and all broader Signed-Agent-Metadata identifier members present in the request. For example, session="sess_1" under project="prj_a" and session="sess_1" under project="prj_b" MUST be treated as distinct sessions.
Omitted Intermediate Tiers: If a platform omits an intermediate member (for example, sending product and session without project), the scoping chain consists of the present members in their defined hierarchical order: (Signature-Agent, product, session).

Identifier Generation, Stability, and Origin Scoping To make Signed-Agent-Metadata effective for isolating noisy neighbors and avoiding shared-fate penalties, platforms SHOULD follow these generation practices:

Platform Attestation and Tenant Isolation: The platform SHOULD populate Signed-Agent-Metadata members from authenticated platform state rather than allowing tenant workloads to supply arbitrary unverified values for project (or for product if shared across tenants). If a platform passes through tenant-supplied session identifiers, the platform SHOULD also include an authenticated project member (or namespace the session identifier per tenant) so that one tenant cannot collide with or spoof identifiers belonging to another tenant.
Identifier Stability:
- product and project values SHOULD remain stable across requests and across signing key rotations for the lifetime of the product or tenant workspace. Generating a new random project value on every HTTP request prevents reputation accrual and reduces the utility of the field.
- session values SHOULD remain constant across HTTP requests belonging to a single bounded task execution, workflow run, or browser context lifecycle, and SHOULD change when a new task or session starts. If a platform rotates session on every request, verifiers cannot link requests within a task and will evaluate traffic at a broader tier.
Origin Scoping for session: To prevent unrelated websites from correlating a session across origins, platforms SHOULD derive session values per target origin (or per registrable domain). For example, a platform can compute an origin-scoped identifier using a keyed hash over the internal session identifier and the target authority:

With origin scoping, a verifier at https://shopping.com observes a consistent session identifier across requests to its own origin, while https://content.org and https://news.net observe different, uncorrelated values.

Signing and Verification

Signing Requests When a platform chooses to send Signed-Agent-Metadata on a signed Web Bot Auth request (), it binds the header field to the HTTP request so verifiers can authenticate it:

Construct the Header: The platform sets the Signed-Agent-Metadata header field with the applicable dictionary members.
Include in Signature-Input: The platform adds "signed-agent-metadata" to the list of covered components in the Signature-Input header field alongside "@authority" and "signature-agent".
Compute the Signature: The platform computes the HTTP Message Signature over the signature base as specified in .

Verifier Processing When receiving an HTTP request containing Signed-Agent-Metadata, a verifier performs the following steps:

Verify Signature Coverage: Check that the Signature-Input header field for the Web Bot Auth signature (tag="web-bot-auth") includes "signed-agent-metadata" in its covered component list. If "signed-agent-metadata" is not covered by the signature, the verifier MUST ignore the Signed-Agent-Metadata header field.
Validate the Signature: Verify the HTTP Message Signature using the public key discovered via Signature-Agent, as specified in . If signature verification fails, the verifier MUST NOT trust the Signed-Agent-Metadata header field.
Parse and Scope: Parse Signed-Agent-Metadata as an Dictionary. If parsing fails, ignore the header field. Otherwise, extract the recognized string members (product, project, session) and construct the Effective Identity tuple scoped under the verified Signature-Agent.

Complete HTTP Request Example The following example shows an HTTP GET request to www.shopping.com, signed by Acme's agent platform (https://platform.acme.com) on behalf of its Acme-Research product, including product, project, and session in Signed-Agent-Metadata (lines folded for readability):

Verifier Considerations (Non-Normative) This section describes how verifiers (including origin servers such as shopping.com, content.org, and news.net, as well as WAFs, CDNs, and bot management services) can use Signed-Agent-Metadata to manage traffic policy and reputation.

Reputation Isolation and Fallback Verifiers can maintain reputation and rate-limiting state at each level of the Effective Identity tuple:

Targeted Enforcement: If a single session (session="sess_3a2b1c") under project="prj_104" exhibits abusive behavior on shopping.com (such as hoarding rocket-powered roller skates or aggressive price scraping), the verifier can rate-limit, challenge, or block that specific session. Other sessions under prj_104 and other projects on platform.acme.com keep their normal standing.
Hierarchical Rollup: If a tenant rotates session identifiers rapidly to evade per-session limits on content.org, the verifier can aggregate anomaly signals at the parent project tier (project="prj_104") and restrict the entire project without penalizing the rest of platform.acme.com.
Graceful Fallback: If a platform signs only product="Acme-Labs" and project="prj_104", the verifier evaluates traffic at the project tier. If a request has no signed Signed-Agent-Metadata header, the verifier falls back to Tier 0 (evaluating all traffic under the platform.acme.com Signature-Agent URI).

Abuse Reporting to Platform Operators When a platform publishes contact endpoints in its Signature Agent Card (contacts in ), a verifier that detects severe abuse can report the incident to the platform by providing:

The Signature-Agent URI and keyid
The exact signed Signed-Agent-Metadata header value
The target origin (@authority) and request timestamp (created)

Because the platform generated the pseudonymous project and session identifiers (or derived session via HMAC with the target origin), the platform can map the reported tuple back to the responsible internal tenant account and take corrective action.

Security Considerations

Unsigned or Stripped Headers An attacker or intermediary could inject a forged Signed-Agent-Metadata header onto an unsigned request or append Signed-Agent-Metadata to a request whose Signature-Input does not cover "signed-agent-metadata". As required in , verifiers MUST ignore Signed-Agent-Metadata unless "signed-agent-metadata" is included in the verified Signature-Input component list. An intermediary could also strip Signed-Agent-Metadata only if "signed-agent-metadata" was not included in Signature-Input. Once "signed-agent-metadata" is listed in Signature-Input, removing or modifying the header invalidates the HTTP Message Signature.

Cross-Tenant Identifier Spoofing If a multi-tenant platform allows customer code to set raw Signed-Agent-Metadata headers directly before the platform's signing proxy signs the outbound request, a malicious customer could set product or project values belonging to another customer. For example, a tenant operating product="Acme-Crimes" could claim product="Acme-Research", project="prj_104" on requests to news.net, either to inherit a trusted customer's reputation or to send abusive traffic that degrades a competitor's standing. To protect its own tenants from cross-tenant spoofing, the signing platform SHOULD overwrite, namespace, or validate Signed-Agent-Metadata at its signing boundary using authenticated tenant metadata ().

Sybil Attacks and Identifier Churn A malicious tenant may attempt to reset a poor reputation by cycling through newly created session or project identifiers on every request. Verifiers SHOULD treat newly observed identifiers as having neutral (unestablished) standing rather than high trust, and SHOULD track the rate of new sub-identifiers created under each parent project, product, or Signature-Agent. High sub-identifier churn under a parent tier is itself a signal to apply policy at the parent tier.

Privacy Considerations

Prohibition of Personally Identifiable Information Signed-Agent-Metadata is transmitted in cleartext HTTP headers to origin servers and intermediaries. Platforms MUST NOT place personally identifiable information (PII)—such as email addresses, legal names, usernames, phone numbers, IP addresses, or billing account names—into project or session. All values for these fields MUST be opaque tokens, UUIDs, or cryptographic hashes.

Cross-Origin Correlation As documented in the Privacy Considerations of , persistent identifiers in signed HTTP requests allow visited origins to correlate client activity across websites. In Signed-Agent-Metadata, product and project identify commercial software or tenant workspaces, whereas session represents an individual task or browsing context. Scoping session identifiers to the target origin () prevents two collaborating origins from linking a session's activity across sites, while still giving each origin a stable handle for local rate limiting and abuse defense.

IANA Considerations

HTTP Field Name Registration IANA is requested to register the following entry in the "Hypertext Transfer Protocol (HTTP) Field Name Registry" defined in :

Field Name:: Signed-Agent-Metadata
Status:: provisional (or permanent if adopted on Standards Track)
Structured Type:: Dictionary
Reference:: This document ()

Open Questions for the Working Group The authors welcome Working Group feedback on the following topics at IETF 127:

Header Field Naming and Future Metadata: This draft proposes Signed-Agent-Metadata (SAM) to indicate that the header field must be covered by the request signature and can accommodate both sub-identifiers (product, project, session) and future per-request metadata (such as session start timestamps or account tenure). Should the Working Group keep Signed-Agent-Metadata, or prefer an alternative such as Agent-Metadata, Signature-Agent-Metadata, Signature-Agent-IDs, Signature-Agent-Scope, or Agent-Context?
Intermediate Identifier Between project and session: Is a fourth identifier needed between project (the customer tenant or workspace) and session (a single task run) to represent a specific deployed agent, sub-tenant, or credential without introducing a human-level user identifier? If so, what term best fits that tier (for example, resource, label, workload, or parent)?
Alignment with the Signature Agent Card Registry: establishes an IANA registry for web_bot_auth metadata parameters. Should the Signed-Agent-Metadata dictionary keys be registered in a sub-registry or shared registry, and should a Signature Agent Card be able to declare which Signed-Agent-Metadata keys the platform signs?
Origin Scoping vs. Cross-Origin Security Providers: Origin-scoped identifiers protect cross-site privacy, whereas security providers operating across multiple origins can reduce cold-start challenges when a pseudonymous identifier is consistent across protected origins. Should the specification define an explicit parameter or convention indicating whether identifiers are origin-scoped or platform-scoped?