terms.txt: Machine Access Terms and an Origin-Enforced Consent and Compensation Exchange

terms.txt: Machine Access Terms and an Origin-Enforced Consent and Compensation Exchange Independent

mailtorajarshi@gmail.com

Web and Internet Transport terms.txt robots.txt Web Bot Auth AI crawlers agents HTTP 402 The Robots Exclusion Protocol lets an origin ask automated clients not to fetch certain paths. It cannot express who is asking, for what purpose, under what terms, or at what price, and it provides no server-side enforcement. This document specifies terms.txt, a file at a well-known location in the style of robots.txt that states, per path and per purpose, whether automated access is allowed, charged, or denied, at what use level, at what price, and whether a user's delegation is required. It also specifies the HTTP exchange that enforces the file at the origin: requests signed with Web Bot Auth, a signed declaration of intent, delegation tokens and payment vouchers bound to the authenticated identifier, payment negotiation signaled with Problem Details, and origin-signed receipts. The document states which properties the exchange enforces before delivery, which it can only audit afterward, and which remain contractual.

Introduction The open web has operated on an implicit bargain: an origin admits crawlers, and search engines return visitors. Automated clients that fetch content to train models or to answer questions directly break that bargain, because the fetch still happens while the referral does not. Origins have responded with the only tool they widely hold, the Robots Exclusion Protocol , which has proven inadequate in four ways. Its vocabulary of allow and disallow cannot distinguish fetching for indexing from fetching for training from fetching to answer a live request. It cannot state a price, a license, or a condition. It depends on the client identifying itself truthfully in a free-text User-Agent field. And it supplies no enforcement, so an origin that wishes to admit an archive but refuse a commercial training crawler has no way to say so and no way to check. Recent work addresses parts of this problem separately. Web Bot Auth lets an automated client sign its requests with HTTP Message Signatures using keys published under an identifier it names, so an origin can verify that the holder of a published key signed the request. The AI Preferences work defines a vocabulary for how content may be used and a way to attach those preferences to content, but by charter it expresses preferences and does not enforce them. Content delivery networks have shipped proprietary mechanisms that classify crawlers by purpose, return HTTP 402 to charged crawlers, and revoke the standing of operators that misbehave, but those mechanisms apply to an origin only because it routes traffic through a particular company. This document composes these pieces into one exchange that any origin can enforce. An origin publishes a terms.txt file () stating its terms per path and per purpose. A client fetches the file, then sends requests that are signed with Web Bot Auth, carry a signed declaration of purpose and use (), and, when the terms require it, carry a delegation token binding the request to a user's entitlements () or a payment voucher (). The origin verifies the signature, checks the declaration against its terms, and either serves the content with a signed receipt () or refuses with a status code and a Problem Details body that states why, including the price and terms when payment is required.

Scope and Enforcement Boundary A signature over a declared purpose proves who made the declaration and that it was not altered in transit. It does not prove that the declaration is true. A signed receipt proves that content was delivered under stated terms. It does not prove what the recipient did with the content afterward. Once content leaves the origin, HTTP cannot govern its use. This document is therefore precise about three tiers. Properties in the first tier are enforced before delivery: the authenticated Web Bot Auth identifier of the requester, the freshness and uniqueness of the request, the binding of the signature to the full target URI, the presence and integrity of the declared intent, the validity and scope of a delegation, the match between the declaration and the terms, and the presence of a valid, unspent payment. Properties in the second tier are auditable after delivery: whether declared purpose matched observed behavior, which the receipt log and the signed declaration make attributable. Properties in the third tier are contractual: what a recipient does with lawfully delivered content, including whether it trains a model on it. This document makes the terms of the third tier explicit, attributable, and priced. It does not make them self-executing, and implementers MUST NOT describe it as doing so.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document applies to requests with the GET and HEAD methods. Requests with other methods are out of scope unless they also satisfy . This document uses the terms "origin", "user agent", "intermediary", "field", and "resource" as defined in , and "Structured Field", "Dictionary", "String", and "Token" as defined in . It uses "operator", "Signature Agent", and "key directory" as defined in .

Agent: An automated HTTP client. This document distinguishes four classes because terms differ by class: a training crawler, which fetches to build a model; a search crawler, which fetches to build an index that refers traffic; a service-operated agent, which fetches in real time on behalf of a service; and a user-delegated agent, which fetches on behalf of a specific person and can inherit that person's entitlements.
Agent identifier: The identifier that results from successful Web Bot Auth resolution of the Signature-Agent member of a request, after the normalization that specifies. Under the default discovery type this is the resolved key directory URL. Tokens and receipts bind to this identifier, not to the raw field value. A valid signature establishes that a holder of a key published under this identifier signed the message; it does not by itself establish who operates the agent.
Purpose: The reason for which an agent declares it is fetching a resource, drawn from the registry in .
Use level: The extent to which the agent declares it will reproduce the fetched content: immediate, reference, or full ().
Terms: The decision, use ceiling, price, and delegation requirement that an origin states for a given path and purpose in its terms.txt file.
Receipt: A signed statement by the origin that a specific resource was delivered to a specific operator under specific terms.
Settlement service: A party that issues payment vouchers an origin will accept. Its relationship to the origin and to the operator is out of scope.
Identity provider: A party that issues delegation tokens on behalf of users. Its relationship to the origin and to the user is out of scope.

Overview of the Exchange Before its first request to an origin, an agent obtains its operator's signing key and, when needed, a delegation token from the user's identity provider and a payment voucher from a settlement service. It MAY fetch the origin's terms.txt file to learn what is required. The exchange then proceeds as follows.

The agent sends an HTTP request carrying a Web Bot Auth signature () and an Access-Intent field declaring purpose and use, together with Access-Delegation and Access-Payment fields when the terms require them. All of these fields are covered by the signature.
The origin verifies the signature, including freshness, replay protection, and coverage of the full target URI, and retrieves the key from the key directory resolved from the Signature-Agent field, caching it under the pair of agent identifier and key identifier.
The origin parses the Access-Intent field and locates the terms for the request path and declared purpose.
If the terms require delegation, the origin validates the Access-Delegation token, including its binding to this agent identifier and its scope.
If the terms charge for access, the origin validates the Access-Payment voucher and spends it atomically. If no valid, unspent voucher is present, it responds with status 403 and a Problem Details body of the "payment-required" type stating the price and terms ().
If every check passes, the origin serves the resource and returns an Access-Receipt field signed with a key it publishes at its own key directory, and appends the receipt to an append-only log.

Unsigned requests, and signed requests that carry no Access-Intent field, are governed per path by the Unsigned directive of the applicable terms: served on paths marked allow, refused with a signature challenge on paths marked challenge, and refused on paths marked deny. No receipt is issued for them.

The terms.txt File

Location and Retrieval An origin publishes its terms at the well-known URI "/.well-known/terms.txt". The resource MUST be served with the media type text/plain and the UTF-8 charset. It SHOULD be cacheable, and origins are encouraged to set a Cache-Control max-age of at least one hour so that clients fetch it infrequently. Origins SHOULD include a Link field with the relation type "access-terms" () on responses to automated clients, pointing at the terms resource, so that a client that did not fetch the file in advance learns where it is. The file applies to the origin at which it is served, in the sense of . It does not apply to other origins, including other ports or schemes at the same host.

Syntax The file is a sequence of lines. Each non-empty, non-comment line is a field name, a colon, and a value, optionally followed by a comment. Field names are case-insensitive. Header fields precede one or more Path blocks. A Path block begins with a Path line and continues until the next Path line or the end of the file. The grammar in ABNF is as follows; the final line of a file MAY omit its line terminator. end = [ WS ] [ comment ] EOL skip = [ WS ] [ comment ] EOL comment = "#" *VCHAR VCHAR-nohash = %x21-22 / %x24-7E WS = 1*WSP EOL = [ CR ] LF ]]> A "#" and everything after it on a line is a comment and is ignored; a path-prefix therefore cannot contain "#". Lines that do not match the grammar MUST cause the entire file to be rejected; an origin whose file fails to parse has stated no terms, and clients MUST treat it as if no terms.txt were present. A header field other than Payment that appears more than once is a parse error.

Header Fields

Version: The version of this specification the file conforms to. This document defines version 1. Files without a Version line are version 1. A client that encounters a Version it does not implement MUST treat the file as absent.
Terms-Id: An opaque identifier for this edition of the terms, chosen by the origin. It is echoed in receipts and payment-required responses so that a receipt can be matched to the terms in force when it was issued. Origins MUST change the identifier whenever the file changes.
Receipt-Keys: The URI of a key directory, in the format defined by , at which the origin publishes the public keys it signs receipts with. A relative reference is resolved against the origin. Origins that issue receipts MUST include this field.
Payment: A payment method the origin accepts, followed by the URI of the settlement service for that method. This document defines the method "voucher" (). The field MAY appear more than once, once per method; each occurrence pairs one method with one settlement URI. Origins that use the charge decision MUST include at least one Payment field.
Purposes: The purpose names the origin recognizes. If absent, the origin recognizes the initial registry contents (). A Purpose line naming a purpose not in this set is a parse error.
Uses: The ordered list of use levels the origin recognizes, from least to most extensive. If absent, the list is "immediate reference full".

Path Blocks A request matches the Path block whose path-prefix is the longest prefix of the request's path component. Before comparison, both the request path and each path-prefix are normalized as in : percent-encoded octets that correspond to unreserved characters are decoded, hexadecimal digits in remaining percent-encodings are uppercased, and percent-encoded reserved characters such as "%2F" are left encoded, since decoding them would change the path's structure. Comparison is then byte-wise. A request that matches no block has no terms and MUST be treated by the origin according to its local policy; this document RECOMMENDS treating it as if the block "Path: /" with "Unsigned: allow" and no Purpose lines applied, which serves unsigned traffic and refuses every declared purpose. Within a block:

Unsigned: The policy for requests that carry no signature. "allow" serves the resource without a receipt. "challenge" refuses with status 403 and an Accept-Signature field (). "deny" refuses with status 403. The default is "allow".
Purpose: The terms for one declared purpose. The decision is one of "allow", "charge", or "deny". A "use=" option sets the maximum use level a client may declare; a declaration above the ceiling is refused. A "delegation=" option requires a valid delegation token whose scope claim includes the named scope. A price is REQUIRED when the decision is "charge" and MUST NOT appear otherwise; its unit names what is being priced, and this document defines the unit "request".

A purpose that has no Purpose line in the matching block is refused. Origins that wish to admit a purpose must say so.

Purpose Names Purpose names are registered in the registry established in . The initial registry contains:

search: Fetching to build an index from which the origin will receive referrals. Analogous to the "search" category of , with one difference: in that vocabulary "search" excludes producing summaries, whereas here the use level, not the purpose, governs how much of the content may be reproduced.
agent: Fetching in real time to satisfy a request from a service or a person, for example to ground an answer.
train-ai: Fetching to train or fine-tune a model. Analogous to the "train-ai" category of .
archive: Fetching to preserve the resource in a public archive.
research: Fetching for non-commercial research.

The last two are included because measurements of robots.txt deployment show that origins reacting to commercial training crawlers block archives and research crawlers as collateral damage, since the Robots Exclusion Protocol cannot distinguish them. Giving them names lets an origin admit them explicitly. The registry is independent of the AI Preferences vocabulary, whose own text notes that it does not yet have working group consensus. Where a name here has an analogue there, the registry entry says so; the names are not claimed to be identical.

Use Levels A use level describes how much of the fetched content the agent declares it will reproduce to its own users. "immediate" means a short excerpt or a fact extracted from the content. "reference" means a summary or a paraphrase with attribution. "full" means reproduction of the content in whole or substantial part. The levels are ordered, and a terms ceiling of a given level admits declarations at that level and below. The three levels are those of a proposed extension to the Content Signals convention described in ; the convention itself defines the categories ai-train, search, and ai-input and does not define use levels. Adopting the extension's names lets an origin that already publishes that signal in robots.txt state enforceable terms in the same words.

Example Under these terms, an unsigned browser is served from /articles/ and challenged on /premium/. A search crawler is admitted everywhere at the reference level. A training crawler pays per request under /articles/ and is refused under /premium/. A user-delegated agent is admitted under /articles/ if its delegation carries the articles scope, and under /premium/ only with the premium scope and a payment voucher.

Request Fields

Access-Intent The Access-Intent request field declares the purpose and use level of the request. It is a Structured Field Dictionary with the following members:

purpose: A String or Token naming a registered purpose. REQUIRED.
use: A String or Token naming a use level. OPTIONAL; the default is "reference".

The field MUST be a covered component of the request signature (). An origin MUST refuse with status 400 a request whose Access-Intent field is present but not covered, since an intermediary could otherwise add or alter the declaration. A signed request that carries no Access-Intent field is processed under the Unsigned directive of the matching Path block, so that clients implementing Web Bot Auth without this document are neither privileged nor broken.

Access-Delegation The Access-Delegation request field carries a delegation token () asserting that the request is made on behalf of a user who holds an entitlement at the origin. The token is issued by an identity provider the origin trusts; how the origin comes to trust an identity provider is out of scope. The field MUST be a covered component of the request signature. A delegation token's claims are:

iss: The identity provider, as an origin.
sub: A pairwise pseudonym for the user, stable for this identity provider and this origin but not correlatable across origins.
aud: The origin the token is valid for.
agent: The agent identifier () permitted to present the token. A token presented in a request that resolves to a different agent identifier MUST be refused.
scope: An array of scope identifiers the user has delegated.
iat, exp: Issuance and expiry times in seconds since the Unix epoch.

Access-Payment The Access-Payment request field carries a payment credential for a charged request. Its value is a method name, a space, and a method-specific credential. This document defines the "voucher" method, whose credential is a token () issued by a settlement service the origin accepts. The field MUST be a covered component of the request signature. A voucher's claims are:

iss: The settlement service, as an origin.
aud: The origin the voucher is valid for.
agent: The agent identifier permitted to spend it.
path: A path prefix the voucher may be spent under.
amount, currency, unit: The value of the voucher and the unit it prices, matching the unit of the terms it is meant to satisfy. Amounts are exact decimal strings, for example "0.01", never JSON numbers. A voucher MUST NOT be accepted for a resource whose price exceeds its amount, whose currency differs, or whose unit differs.
id: A unique identifier. An origin MUST accept a given identifier at most once, and MUST check and record the identifier as spent in one atomic operation so that two requests presenting the same voucher concurrently cannot both succeed.
exp: Expiry time in seconds since the Unix epoch.

How the settlement service is funded, how it settles with the origin, and whether the unit of account is the request, the citation, the answer, or the transaction are out of scope. The receipt defined in is designed to make any of those units measurable.

Signature Requirements Requests that carry an Access-Intent field MUST be signed as specified in . This document profiles that specification as follows:

The covered components MUST include "@method", "@target-uri", the "signature-agent" field member keyed by the signature label, and "access-intent". They MUST also include "access-delegation" and "access-payment" whenever those fields are present. Covering "@target-uri" rather than "@path" binds the scheme, authority, path, and query; a signature that covered only the path would let an intermediary or attacker alter the query without invalidating it.
Requests with a method other than GET or HEAD MUST also carry a Content-Digest field and cover it.
The signature parameters MUST include "created", "expires", "nonce", "keyid", and tag="web-bot-auth". The interval between "created" and "expires" MUST NOT exceed 300 seconds.
Signing keys MUST be Ed25519 keys. The "alg" parameter MAY be omitted; if present, its value MUST be "ed25519", the identifier registered for HTTP Message Signatures, which is distinct from the JOSE identifier used in .
"keyid" MUST be the JWK Thumbprint of the signing key, computed as specified for OKP keys in .

An origin MUST look keys up by the pair of agent identifier and key identifier, and MUST NOT accept a key found under one identifier's directory for a request that resolves to another.

Response Fields and Status Codes

Access-Receipt On every response that serves a resource to a signed request, the origin MUST include an Access-Receipt field whose value is a token () signed with a key published at the origin's Receipt-Keys directory. Its claims are:

iss: The origin.
kid: The thumbprint of the signing key.
agent: The agent identifier served.
sub: The subject pseudonym from the delegation token, or null.
purpose, use: The declared purpose and use level.
terms_id, terms_digest: The Terms-Id in force and the base64url-encoded SHA-256 digest of the terms.txt representation it names, so that the receipt proves what the terms contained and not only what they were called.
method, target: The request method and the full target URI.
status: The response status code.
content_digest: The value of the response's Content-Digest field , binding the receipt to the representation delivered. Origins that issue receipts MUST include a Content-Digest field on receipted responses.
req: The base64url-encoded SHA-256 hash of the request's Signature field value, binding the receipt to one signed request.
ts: Issuance time in seconds since the Unix epoch.
id: A unique identifier.

The origin SHOULD append each receipt to an append-only log whose entries are hash-chained, so that the log head commits to every receipt issued. The receipt gives both parties a non-repudiable record of which representation was delivered to which agent identifier under which terms, which is the input any settlement model or dispute requires. A receipt is evidence for the one request whose Signature field it hashes and for no other; in particular, a receipt obtained from a cache is not a receipt for the request that hit the cache ().

The "access-terms" Link Relation A Link field with relation type "access-terms" identifies the origin's terms.txt resource. Origins SHOULD include it on 400 and 403 responses to automated clients and MAY include it on 200 responses.

Status Codes This document uses existing HTTP status codes with their defined semantics and conveys the specific reason in a Problem Details body , in accordance with BCP 56 . It does not use 401, which requires a WWW-Authenticate field that no HTTP authentication scheme here would supply, and it does not use 402, which RFC 9110 reserves and whose semantics this document is not the place to define.

200: The request satisfied the terms. The response carries an Access-Receipt field.
400: The request's signature fields or Access-Intent field are malformed, required signature parameters are absent, or a field that this document requires to be covered is present but not covered.
403: The request is refused. The Problem Details "type" states why: the request is unsigned on a path marked challenge or deny; the signature could not be verified (unknown key, signature over a different message, expired or stale); the nonce was already used; the purpose is denied or unknown, or the declared use exceeds the ceiling; a required delegation is absent, invalid, expired, out of scope, or bound to another agent identifier; or the terms charge for this purpose and no valid, unspent payment was presented. An origin MAY answer repeated replays from one agent identifier with 429 instead.

Every 403 response under this document MUST carry a Problem Details body of media type application/problem+json and Cache-Control: no-store. For payment, the "type" member is the "payment-required" problem type registered in , and the body carries these additional members: "terms_id"; "purpose"; "price", an object with "amount" (an exact decimal string), "currency", and "unit"; "payment", an array of objects each with "method" and "settlement", so that each method is paired with its settlement URI; and "terms", the URI of the terms resource. For other refusals the "type" member is the "terms-refused" problem type and the body SHOULD carry "reason", one of "unsigned", "signature-invalid", "replay", "purpose-refused", "use-exceeds-terms", or "delegation-required". Deployed systems that charge crawlers have used status 402 for the payment case. This document does not depend on that usage. Should the HTTP Working Group define semantics for 402, a future version of this document can adopt them.

Signature Challenge When an unsigned request arrives on a path whose Unsigned directive is "challenge", the origin responds with status 403, a Problem Details body with reason "unsigned", and an Accept-Signature field as defined in listing the components and parameters the origin requires, which is the behavior recommends for requesting a signature. For example:

Token Format Delegation tokens, payment vouchers, and receipts are JSON Web Signatures in Compact Serialization signed with Ed25519 . The JOSE header MUST contain "alg" with the value "Ed25519", the fully specified identifier registered by , and "kid" with the thumbprint of the signing key. The deprecated polymorphic identifier "EdDSA" MUST NOT be used. The payload is the JSON object of claims . Verifiers MUST reject a token whose "alg" is anything else, whose signature does not verify under the key identified by "kid" at the expected issuer, or whose "exp" claim is in the past. Verifiers MUST enforce audience and agent binding. A token whose "aud" is not this origin, or whose "agent" is not the agent identifier that the request's Signature-Agent resolved to, MUST be refused. These two checks are what make a stolen token useless at any other origin or under any other identifier.

Origin Key Publication An origin that issues receipts publishes the corresponding public keys at the directory named by Receipt-Keys, using the same directory format and the same thumbprint key identifiers as . A third party verifying a receipt fetches the directory, selects the key whose thumbprint matches the receipt's "kid", and verifies the signature. Reusing the Web Bot Auth directory format means origins and operators publish keys the same way.

Processing Rules

Origin On receiving a request for a resource governed by terms.txt, an origin proceeds in the following order. The order matters for security: inexpensive checks precede cryptography, and replay reservation precedes any asynchronous discovery.

Locate the Path block for the request. If the request is unsigned, or signed but without an Access-Intent field, apply the Unsigned directive and stop.
Parse the Signature-Input field. Refuse with 400 if the tag, "created", "expires", "nonce", or "keyid" parameters are absent or malformed, if an "alg" parameter is present with a value other than "ed25519", or if the components required by are not covered. Refuse with 403 if the signature is expired or stale.
Resolve the Signature-Agent member to an agent identifier as specifies. Atomically check and reserve the tuple of agent identifier, "keyid", and "nonce" in a replay store scoped to at least the signature validity window. If the tuple is already present, refuse with 403 (or 429). The reservation MUST happen before any network operation, so that two identical requests racing a cold key lookup cannot both succeed. If a later step fails, the reservation MAY be released.
Obtain the key, from cache or by fetching the directory. Fetches MUST be bounded in size and time, MUST NOT follow redirects, and SHOULD be coalesced so that concurrent requests naming one directory cause one fetch. Verify the signature; refuse with 403 on failure.
Parse Access-Intent. Refuse with 400 if it is not covered by the signature. Locate the Purpose line; refuse with 403 if there is none, if the decision is deny, or if the declared use exceeds the ceiling.
If the terms require delegation, validate the Access-Delegation token as specified in and , including scope; refuse with 403 on failure.
If the decision is charge, validate the Access-Payment voucher and atomically check and record its identifier as spent; respond with 403 and the "payment-required" problem type if it is absent, invalid, insufficient, of the wrong unit, expired, or already spent.
Serve the resource with Content-Digest and Access-Receipt fields and append the receipt to the log.

Client A client SHOULD fetch and cache the origin's terms.txt before its first governed request, and MUST refetch it when a 403 response carries a Link with the "access-terms" relation and the client's cached copy is older than the response. A client MUST declare a purpose it can substantiate and a use level it will not exceed. A client MUST NOT retry a refused request with a different declared purpose for the same fetch in order to find one the terms admit; it MAY select a purpose from the terms before the request. A client SHOULD verify the Access-Receipt it receives against the origin's published key and retain it, since the receipt is its evidence of the terms it was served under.

Caching and Intermediaries A receipt is evidence for exactly one signed request. A response stored by any cache , shared or private, and reused for a later request would carry a receipt that hashes a different request's signature, and the origin would have issued no receipt and logged nothing for the later one. Origins MUST therefore mark responses that carry an Access-Receipt with Cache-Control: no-store. Responses to unsigned requests are unaffected and remain cacheable as today. A future version of this document may define a cacheable variant in which the receipt is obtained separately from the representation. Because the signature covers the full target URI and the request fields defined here, an intermediary in front of the origin may forward the request unchanged for the origin to verify, or may verify on the origin's behalf and convey the result by a mechanism outside this document. An intermediary that rewrites a covered component invalidates the signature, which is the intended failure.

Security Considerations The security considerations of and apply. The following are specific to this document. False declarations. Nothing in this document prevents an operator from declaring a purpose or use level that does not match its behavior. What the signed declaration provides is attribution: an operator that declares reference use and reproduces content in full, or declares search and never refers, leaves evidence in receipts, in its own outputs, and in traffic, and that evidence names the operator. The remedy is revocation of the operator's standing, which is outside this document. Target binding. A signature that covered only the path would leave the query string unbound, so that a request for one resource could be replayed against another at the same path. Covering "@target-uri" closes this. Requests with bodies are out of scope unless the body is covered through Content-Digest, since an uncovered body is likewise alterable. Credential theft. Signature validity is bounded to 300 seconds. Delegation tokens and vouchers are bound to an audience and an agent identifier, so a token stolen from one agent cannot be presented under another identifier. Key rotation is handled through the key directory. Replay. The replay store is the only defense against reuse of a captured signed request within its validity window. It MUST be consulted and updated atomically, before any asynchronous operation, and in multi-instance deployments MUST be shared across instances. A first version of the reference implementation reserved the nonce after key discovery and was vulnerable to two identical requests racing a cold lookup. Cost-based denial of service. Verification costs the origin CPU per request and an attacker can force it. Origins SHOULD order cheap checks before cryptography, as in , SHOULD rate-limit per Signature Agent URL, and can rely on the observation that refusing a request costs less than serving one. In the reference implementation on a single shared vCPU, rejecting a forged signature cost 0.21 ms and a payment refusal 0.37 ms, against 0.65 ms for a paid, receipted response. Identity. A verified signature establishes that a holder of a key published under the resolved agent identifier signed the request. It does not establish who operates the agent, and an origin that needs a real-world identity behind an identifier must obtain it by other means, for example the operator registry that anticipates. This document's terms, tokens, and receipts bind to the identifier, and its guarantees are stated in those terms. Directory fetching. Fetching an operator's key directory is a server-side request to a URL named by the client. Origins MUST apply the discovery protections of , including bounding response size and time, refusing redirects, and refusing non-HTTPS or non-origin URLs. Unauthenticated clients. This document does not defeat scraping by clients that do not sign. It makes signed access more capable than unsigned access, since only signed requests can obtain delegated entitlements, paid content, or receipts, and it lets an origin require signatures on the paths where it matters.

Privacy Considerations The exchange reveals the operator's identity to the origin on every request; that is its purpose. It is designed not to reveal the user's identity. The delegation token's subject is a pairwise pseudonym chosen by the identity provider, stable for one identity provider and one origin, so that an origin can recognize a returning subscriber's agent without learning who the subscriber is or correlating them across origins. Receipts carry the pseudonym and no other user identifier. Identity providers MUST NOT use a globally stable identifier as the subject. The terms.txt file itself is public and reveals an origin's pricing and policy. Origins that consider that sensitive can state terms only for the purposes they wish to admit.

IANA Considerations

Well-Known URI Registration IANA is requested to register the following in the "Well-Known URIs" registry :

URI suffix: terms.txt
Change controller: IETF
Specification document: This document, .
Status: permanent

On the name: discourages names that are overly generic. "terms.txt" is proposed because the resource is a text file of access terms and because the name signals, to the site operators who must deploy it, continuity with robots.txt, which is the mechanism it extends. Should the designated expert prefer a more specific name, "access-terms.txt" preserves both properties.

HTTP Field Name Registrations IANA is requested to register the following in the "Hypertext Transfer Protocol (HTTP) Field Name Registry" , each with status "permanent" and this document as reference: Access-Intent (), Access-Delegation (), Access-Payment (), and Access-Receipt ().

Link Relation Registration IANA is requested to register the following in the "Link Relation Types" registry :

Relation name: access-terms
Description: Refers to a resource stating the terms under which automated clients may access the context resource.
Reference: This document, .

HTTP Problem Type Registrations IANA is requested to register the following in the "HTTP Problem Types" registry :

Type URI: https://iana.org/assignments/http-problem-types#payment-required
Title: Payment required by access terms
Recommended HTTP status code: 403
Reference: This document, .

Type URI: https://iana.org/assignments/http-problem-types#terms-refused
Title: Refused by access terms
Recommended HTTP status code: 403
Reference: This document, .

terms.txt Purpose Names Registry IANA is requested to establish a "terms.txt Purpose Names" registry with the "Specification Required" policy . Each entry has a name, a description, and a reference. The initial contents are the five names defined in , with this document as reference. Designated experts SHOULD keep names aligned with the categories of where the meanings coincide.

Implementation Status This section records the status of known implementations as described in . It will be removed before publication as an RFC. A reference implementation in dependency-free JavaScript on Node.js 22 is archived at and developed at https://github.com/rch0wdhury/terms-txt under the MIT license. It implements the terms.txt parser, the origin processing rules of , a directory resolver, and a harness of 24 conformance and correctness checks, including verification of the Ed25519 test vector E.2.1 of , sequential and concurrent replay rejection, pair-scoped key lookup, scope and agent binding of delegation tokens, single-use vouchers, per-path unsigned policy, and hash-chain verification of the receipt log. Measured on one shared vCPU over loopback, identity verification added 0.20 ms per request over a passthrough server, a receipted search request 0.34 ms, a delegated request 0.49 ms, a payment refusal 0.37 ms, and a paid receipted request 0.65 ms. Release v0.1 predates this document and deviates from it in the following respects, all of which are scheduled for the next release: it covers "@path" rather than "@target-uri"; it answers unverifiable signatures with 401 and payment with 402 rather than 403 with Problem Details; its tokens use a two-part payload-and-signature form rather than JWS with "alg" "Ed25519"; it binds tokens and receipts to the raw Signature-Agent value rather than the resolved agent identifier; its receipts omit the response status, Content-Digest, and terms digest; its vouchers omit the unit and encode amounts as numbers; it marks receipted responses private rather than no-store; and its Link relation is named "terms". Its measurements are reported for that release.

References Normative References Key words for use in RFCs to Indicate Requirement Levels Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words Uniform Resource Identifier (URI): Generic Syntax Augmented BNF for Syntax Specifications: ABNF The Web Origin Concept JSON Web Signature (JWS) JSON Web Key (JWK) Thumbprint Edwards-Curve Digital Signature Algorithm (EdDSA) CFRG Elliptic Curve Diffie-Hellman (ECDH) and Signatures in JSON Object Signing and Encryption (JOSE) Guidelines for Writing an IANA Considerations Section in RFCs The JavaScript Object Notation (JSON) Data Interchange Format Web Linking Well-Known Uniform Resource Identifiers (URIs) HTTP Semantics HTTP Caching HTTP Message Signatures Problem Details for HTTP APIs Digest Fields Fully-Specified Algorithms for JOSE and COSE Structured Field Values for HTTP HTTP Message Signatures for automated traffic Informative References Building Protocols with HTTP Your site, your rules: new AI traffic options for all customers Improving Awareness of Running Code: The Implementation Status Section Robots Exclusion Protocol A Vocabulary for Expressing AI Usage Preferences Associating AI Usage Preferences with Content in HTTP Content Signals PolicyContent Signals terms.txt v0.1: reference implementation and benchmark results

Complete Example Exchange A user-delegated agent operated by https://bot.example requests a resource under /premium/ at origin.example, whose terms are those of . Line breaks in field values are for presentation only. Had the request carried no voucher, the origin would have answered: ; rel="access-terms" {"type": "https://iana.org/assignments/http-problem-types#payment-required", "title":"Payment required by access terms","status":403, "terms_id":"2026-09-01","purpose":"agent", "price":{"amount":"0.01","currency":"USD","unit":"request"}, "payment":[{"method":"voucher", "settlement":"https://settle.example/.well-known/settlement"}], "terms":"/.well-known/terms.txt"} ]]> With a valid voucher, the origin serves the resource: ; rel="access-terms" Access-Receipt: eyJhbGciOiJFZDI1NTE5Iiwia2lkIjoiLi4uIn0.eyJpc3MiOiJv... ... ]]> The receipt payload, decoded:

Acknowledgments This document composes work by the authors of HTTP Message Signatures, Web Bot Auth, AI Preferences, and the Content Signals convention, and draws on measurements published by Cloudflare, Imperva, the Pew Research Center, and the Data Provenance Initiative. The enforcement boundary in and the replay ordering in were sharpened by review of an earlier version of the reference implementation.