| Internet-Draft | LLM Context File Discovery | September 2026 |
| Arsentev | Expires 15 March 2027 | [Page] |
Publishers have begun to serve a curated, plain-text summary of a web origin intended for consumption by large language models and by the crawlers that feed them, most visibly under the de facto file name "llms.txt". The practice has no specification, no media type, and — of direct operational consequence — no discovery mechanism: a consumer that does not already guess the path cannot learn that such a file exists.¶
This document specifies discovery and retrieval for publisher-curated context files. It defines the well-known URI "llm-context", the link relation type "llm-context", and an extension record for the robots exclusion protocol, so that a publisher may advertise a context file by three independent paths and a consumer may find it without guessing. It specifies a two-tier arrangement of an index resource and optional detail resources, states conditional-request and size requirements that keep retrieval affordable for both parties, and describes the relationship of this mechanism to the robots exclusion protocol, to sitemaps, and to work in progress on expressing AI usage preferences.¶
This document also reports measurements from an operational deployment in which twenty crawlers operated by search and language-model providers issued 44,005 requests to a host over fifteen days without once retrieving the context file the host was serving, while the same crawlers retrieved that host's robots.txt 577 times in the three days after the context file was deployed. The absence of a discovery mechanism, rather than the absence of interest, is the hypothesis this document acts upon.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 15 March 2027.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
A web origin that wishes to be represented accurately to a large language model (LLM) faces a problem that ordinary search engine optimisation does not solve. The origin's HTML is written for a browser: it carries navigation, boilerplate, advertising, and markup whose only purpose is presentation. A consumer that ingests that HTML in order to build a representation of the origin must strip it, and the stripping is lossy and unsupervised. The publisher, who knows which pages matter and how the origin should be described, has no channel through which to say so.¶
Beginning in 2024 a convention emerged in which publishers serve a plain-text, Markdown-formatted summary of the origin at the fixed path "/llms.txt", frequently accompanied by a larger concatenation at "/llms-full.txt". The convention is widely deployed and widely discussed. It is also entirely unspecified: there is no media type, no statement of what a conforming consumer is required to do, no size discipline, and no way for a consumer to learn that the file exists other than to request a path it has guessed.¶
Guessing is the weak point. The web already has two standardised mechanisms by which an origin tells an automated consumer what it may and should retrieve — the robots exclusion protocol [RFC9309] and the Sitemap protocol — and automated consumers use both routinely. A third convention that is not reachable through either of them is invisible to any consumer that has not been specifically programmed for it.¶
Section 2 reports what that invisibility looks like in an operational deployment. In summary: on a host serving a context file, crawlers operated by search and language-model providers issued 13,917 requests over the three days following deployment, of which 577 were for robots.txt and 249 for sitemap.xml, and none at all were for the context file.¶
This document does not attempt to standardise the content of a context file, which is a matter of editorial judgement and is likely to remain so. It specifies the three things that must be common for the mechanism to work at all:¶
The registrations required to make the mechanism usable are requested in Section 8.¶
This document is Informational. It describes a mechanism and requests the IANA registrations that the mechanism needs. It does not claim consensus of the IETF.¶
The following are explicitly out of scope:¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
The design in this document is a response to a measurement rather than to an intuition. This section reports the measurement, its method, and its limitations. It is presented here, before the specification, because the specification is difficult to justify without it.¶
A single publisher operates two hosts, "arsentev.ai" and "ru.arsentev.ai", serving the same content in two languages. Both hosts began serving a context file at "/llms.txt" and a detail resource at "/llms-full.txt" on 2026-09-09. Both files are generated from the same source as the HTML pages and are rebuilt on deployment.¶
The access logs of the origin server were parsed for the period 2026-08-28 to 2026-09-11 inclusive, a window of fifteen days containing 331,758 logged requests. Requests were attributed to a crawler when the request's User-Agent header contained one of twenty-two tokens published by search and language-model operators as identifying their crawlers. Attribution by User-Agent is self-asserted and unverified; Section 2.3 returns to this.¶
The sizes of the files as served, in octets, were:¶
| Host | /llms.txt | /llms-full.txt |
|---|---|---|
| arsentev.ai | 29,146 | 367,587 |
| ru.arsentev.ai | 38,982 | 614,939 |
Over the full fifteen-day window, twenty distinct crawler tokens were observed, accounting for 44,005 requests. The number of those requests that retrieved "/llms.txt" or "/llms-full.txt" was zero.¶
Restricting attention to the three days after the context files were deployed, 2026-09-09 to 2026-09-11, so that the crawlers had something to retrieve:¶
| Quantity | Value |
|---|---|
| Requests attributed to crawlers | 13,917 |
| Distinct crawler tokens | 18 |
| Retrievals of /robots.txt | 577 |
| Distinct crawlers retrieving /robots.txt | 12 |
| Retrievals of /sitemap.xml | 249 |
| Retrievals of /llms.txt or /llms-full.txt | 0 |
The comparison that matters is internal to the table. The same crawlers, in the same window, on the same host, retrieved the two resources that a standard tells them to retrieve — robots.txt, defined by [RFC9309], and sitemap.xml — several hundred times, and retrieved the resource that no standard mentions zero times. The crawlers were present, were active, and were willing to fetch metadata about the origin. What they had no way to learn was that a third piece of metadata was on offer.¶
The following bound what may be concluded. They are stated in full because the measurement is a single observation and the temptation to over-read it is real.¶
The design response to this uncertainty is to make the mechanism cheap for a consumer to ignore: every discovery path defined in Section 4 is additive, none changes the meaning of an existing record, and a consumer that implements none of them sees the origin exactly as it does today.¶
A context file MUST be a UTF-8 encoded text document. It SHOULD be formatted as Markdown, and a consumer MUST NOT fail to process a context file that is not valid Markdown; the format is a convenience for human maintainers and a hint for consumers, not a validation gate.¶
A context file MUST NOT be served with a "Content-Type" of "text/html". Publishers that generate the file through an HTML template routinely misconfigure this, and a consumer that receives HTML cannot distinguish a context file from an error page.¶
A context file SHOULD be served as "text/markdown;charset=UTF-8" [RFC7763] or as "text/plain;charset=UTF-8". The registration of a distinct media type is discussed in Section 8.4 and is deliberately not requested by this document.¶
Deployed practice separates a short summary from a long concatenation of full text, and the separation is load-bearing. Table 1 shows the two differing by a factor of twelve on one host and sixteen on the other; the detail resource on one host exceeds 600 kilobytes, and it grows with the site without bound.¶
Accordingly:¶
The size guidance is a floor for interoperability, not a recommendation to fill it. A consumer MAY impose its own limits and MUST fail safely — by treating the origin as having no context file — rather than truncate mid-document in a way that changes meaning.¶
A publisher serving an origin in several languages SHOULD serve a separate context file per origin where languages occupy separate origins, as in the deployment of Section 2, and MAY use proactive content negotiation on a single origin. Where negotiation is used, the response MUST carry an appropriate "Content-Language" and MUST include "Accept-Language" in "Vary".¶
Three mechanisms are defined. They are independent: a publisher MAY deploy any subset, and a consumer MAY implement any subset. A publisher SHOULD deploy at least the well-known URI of Section 4.1, because it is the only one of the three that a consumer can use without first retrieving another resource.¶
Where more than one mechanism is present and they disagree, a consumer MUST resolve the conflict as specified in Section 4.4.¶
This document registers the well-known URI suffix "llm-context" (Section 8.1), following [RFC8615].¶
A publisher advertising a context file through this mechanism MUST arrange that a GET request for "/.well-known/llm-context" on the origin returns either:¶
A consumer SHOULD follow at most five redirects and MUST treat a redirect chain that leaves the origin as requiring the same scrutiny as any other cross-origin reference; see Section 7.2.¶
A 404 response MUST be interpreted as "this origin does not advertise a context file by this mechanism" and MUST NOT be interpreted as an error to be retried aggressively. A consumer SHOULD NOT re-probe a 404-returning origin more often than once per 24 hours.¶
The well-known path is preferred over the deployed convention of a root-level "/llms.txt" for the reason [RFC8615] gives generally: the root namespace belongs to the publisher, and a specification that claims a root path claims something it does not own. A publisher that already serves "/llms.txt" can satisfy this document with a single redirect and need not move the file.¶
This document registers the link relation type "llm-context" (Section 8.2).¶
A publisher MAY advertise a context file by including a "Link" header field [RFC8288] in responses from the origin:¶
Link: </llms.txt>; rel="llm-context"; type="text/markdown"¶
or, in an HTML document, by a link element in the document head:¶
<link rel="llm-context" href="/llms.txt" type="text/markdown">¶
A publisher using the header form SHOULD emit it on responses for the origin's home document and SHOULD NOT emit it on every response, since the field costs bytes on each response and carries no per-response information.¶
The "type" parameter, when present, is a hint. A consumer MUST rely on the "Content-Type" of the retrieved representation and not on the hint.¶
This mechanism exists because it is the only one of the three that works for a consumer which has already retrieved a page and has no reason to make a second request speculatively. An agent fetching a single document on a user's behalf (Section 5.3) is in exactly that position.¶
A publisher MAY advertise a context file by including a record in the origin's robots.txt:¶
User-Agent: * Allow: / LLM-Context: https://example.com/llms.txt¶
The record's name is "LLM-Context", compared case-insensitively as [RFC9309] requires of record names. Its value MUST be an absolute URI. A relative reference MUST be ignored rather than resolved, so that a parser error cannot cause a consumer to request an unintended path.¶
Section 2.2.4 of [RFC9309] permits a crawler to interpret records that are not part of the robots exclusion protocol, giving "Sitemaps" as its example, and requires that the parsing of such records "MUST NOT interfere with the parsing of explicitly defined records". This record is therefore safe to deploy: a crawler that has never heard of it parses the rest of the file exactly as before. That property is the reason for defining it despite the well-known URI already existing — robots.txt is retrieved by consumers today, as Table 2 shows, and a publisher can therefore reach them through it without waiting for new code to be deployed on the consumer side. The record is nevertheless the weakest of the three mechanisms, because it depends on parsers being as forgiving in practice as the specification requires.¶
A record appearing outside any group applies to the origin as a whole. A publisher SHOULD place it outside all groups. A consumer encountering the record inside a group MUST treat it as applying to the origin as a whole, matching the handling Section 2.2.4 of [RFC9309] gives to the "Sitemaps" record, which "MUST NOT terminate a group".¶
A publisher MUST NOT use this record to advertise a context file whose retrieval the same robots.txt disallows. A consumer encountering that contradiction MUST honour the exclusion and MUST NOT retrieve the file: a permission mechanism and a discovery mechanism disagreeing is resolved in favour of the permission mechanism, always.¶
Where several mechanisms advertise different URIs, a consumer MUST apply the following precedence, highest first:¶
A consumer MUST NOT retrieve more than one index resource per origin per retrieval cycle merely because several mechanisms are present.¶
All three mechanisms are subordinate to the robots exclusion protocol as a permission mechanism: a consumer MUST evaluate the exclusion rules of [RFC9309] against the index resource's URI before retrieving it, regardless of which mechanism advertised it.¶
A publisher SHOULD serve context files with a strong validator — an "ETag", or a "Last-Modified" where an entity tag is not available — as specified in [RFC9110].¶
A consumer that has previously retrieved a context file MUST issue a conditional request on subsequent retrievals, using "If-None-Match" where it holds an entity tag and "If-Modified-Since" otherwise.¶
This requirement is not a nicety. A detail resource of the size reported in Table 1, retrieved unconditionally by eighteen crawlers on the cadence at which those crawlers retrieved robots.txt in Table 2, would transfer more data per day than the rest of the origin combined. A mechanism that is expensive to support will not be supported.¶
A publisher SHOULD set an explicit "Cache-Control" "max-age" on context files. A consumer SHOULD NOT retrieve an index resource more often than once per hour, or a detail resource more often than once per day, in the absence of an explicit shorter freshness lifetime.¶
A consumer SHOULD offer content coding in "Accept-Encoding", and a publisher SHOULD serve context files compressed where the consumer offers it. Context files are highly compressible prose and the saving is substantial at the sizes involved.¶
Two kinds of consumer are in view and they have opposite cost profiles. A crawler retrieves on its own schedule, in bulk, and can afford a detail resource. An agent acting on a user's immediate request retrieves one document, once, under a latency budget measured in seconds, and cannot.¶
The asymmetry is not merely one of politeness. In an agent that retransmits its accumulated context to a model on every step, material admitted once is paid for on every subsequent step of the run, so the cost of retrieving a 600-kilobyte detail resource is not the transfer but its persistence in the context window; [I-D.arsentev-agent-run-metrics] specifies an interchange format for accounting of exactly that effect.¶
Accordingly an on-demand consumer SHOULD retrieve only the index resource, and SHOULD NOT retrieve a detail resource unless the user's request cannot be served from the index. A publisher SHOULD therefore write the index resource so that it is useful alone, rather than as a table of contents that is meaningless without its targets.¶
[RFC9309] answers the question "may this consumer retrieve this URI". This document answers the question "which URI should this consumer retrieve". The questions are disjoint and this document does not alter the answer to the first. A context file is subject to exclusion like any other resource, and Section 4.3 and Section 4.4 both resolve contradictions in favour of exclusion.¶
A sitemap enumerates URIs and carries no judgement about them: it is complete by intention. A context file is curated and selective by intention. A publisher SHOULD serve both, and SHOULD NOT generate one from the other mechanically, as a context file that is a transcription of a sitemap communicates nothing that the sitemap did not.¶
Work is in progress in the IETF on vocabularies for expressing preferences about the use of content by AI systems, and on attaching those preferences to content [I-D.ietf-aipref-vocab] [I-D.ietf-aipref-attach]. That work and this document are orthogonal and complementary: the former concerns what a consumer is permitted to do with material it has obtained, the latter concerns how a publisher offers material in the first place.¶
The two interact in one direction only, and the direction matters. Serving a context file MUST NOT be construed as granting any usage preference, relaxing any preference expressed by other means, or waiving any right. A consumer MUST evaluate usage preferences independently of discovery. A publisher that both restricts usage and serves a context file is making a coherent statement — "if you are permitted to use this origin at all, use this description of it" — and a consumer MUST NOT treat the second half as withdrawing the first.¶
Structured data embedded in pages describes entities in machine terms. A context file describes an origin in prose to a consumer whose native input is prose. They are not substitutes, and a publisher SHOULD continue to serve structured data unchanged.¶
This is the consideration that matters most, and it is a consequence of the mechanism's purpose rather than of any detail of its design.¶
A context file is written by a publisher for the express purpose of being placed in the input of a language model. The publisher chooses every word. A file may therefore contain text crafted to be read by the model as instruction rather than as content — directions to disregard prior instructions, to represent the origin in terms the publisher prefers, to disparage a competitor, or to emit a particular URI in an answer. Nothing in this document, and nothing that could be added to it, prevents this: the file is by definition attacker-chosen text from the point of view of any consumer that does not trust the publisher.¶
Therefore a consumer MUST treat the content of a context file as untrusted data and MUST NOT treat any part of it as instruction. A consumer that incorporates a context file into a model's input MUST mark its provenance in a way that survives into the model's input, so that content originating from an origin is distinguishable from the consumer's own instructions. Consumers SHOULD apply whatever isolation they apply to other retrieved web content; a context file deserves no more trust than an arbitrary HTML page from the same origin, and its curated presentation may make it appear to deserve more.¶
A consumer MUST NOT follow URIs found in a context file automatically without applying the same policy it applies to other discovered URIs, including exclusion rules and cross-origin scrutiny.¶
A discovery mechanism that names a URI on another origin allows one party to make assertions about a resource it does not control, and allows an origin to direct consumers elsewhere.¶
A consumer MUST NOT attribute the content of a cross-origin index resource to the advertising origin. A consumer SHOULD decline to retrieve a cross-origin index resource at all unless it has a specific reason to accept the indirection, and MUST evaluate the target origin's own exclusion rules before retrieving it.¶
A publisher can serve a context file that does not correspond to what a browser is shown, which is the long-standing practice known as cloaking, applied to a new surface. This document cannot prevent it. A consumer that relies on a context file SHOULD sample the origin's ordinary representations and SHOULD reduce its reliance on a context file that systematically disagrees with them.¶
The size limits of Section 3.2 and the conditional request requirements of Section 5.1 exist partly for safety. A publisher can otherwise advertise an arbitrarily large resource to every consumer that implements this mechanism, and a consumer that retrieves unconditionally can otherwise impose unbounded cost on a publisher. A consumer MUST impose its own ceiling on the size of any retrieved context file and abandon a retrieval that exceeds it.¶
A request for a well-known URI is as identifying as any other request from the same client and carries no additional information about a user. A publisher can, however, use a distinct path per consumer to determine which consumers implement this mechanism; this is the ordinary observability of an HTTP request and is noted here only because the well-known path is otherwise uniform.¶
A publisher MUST NOT place in a context file personal data that it would not place in a public HTML page. The file is world-readable, and its purpose is to encourage copying of its contents into corpora from which withdrawal is not practical.¶
IANA is requested to register the following entry in the "Well-Known URIs" registry established by [RFC8615]:¶
IANA is requested to register the following entry in the "Link Relation Types" registry established by [RFC8288]:¶
[RFC9309] does not establish an IANA registry of record names, so no registration is requested. The record name "LLM-Context" is documented in Section 4.3 of this document. Should a registry be established, this document requests that "LLM-Context" be entered in it with this document as reference.¶
No media type registration is requested. A context file is Markdown or plain text and the registrations of [RFC7763] and [RFC2046] are adequate. A distinct media type would allow a consumer to recognise a context file retrieved without discovery, but at the cost of a registration whose only distinguishing property is the intent of its author. The trade-off is noted here so that it can be revisited if operational experience favours the other side.¶
This section records the status of known implementation in accordance with [RFC7942]. It is to be removed before publication as an RFC, should that occur.¶
Publisher-side, arsentev.ai and ru.arsentev.ai. Two production hosts serve an index resource at "/llms.txt" and a detail resource at "/llms-full.txt", generated from the same source as the HTML pages, since 2026-09-09. Sizes are given in Table 1. Coverage of this document at the time of writing is partial: the two-tier arrangement of Section 3.2 is implemented and the index resource is within the size guidance on both hosts; the detail resource on ru.arsentev.ai is within the division threshold but approaching it; the discovery mechanisms of Section 4 are not yet deployed, since this document defines them. The operator is the author, and this is therefore a report of one deployment rather than of independent interoperation.¶
Consumer-side. No consumer implementation of the discovery mechanisms defined here is known. Section 2 reports that eighteen crawlers observed over the measurement window implemented none of them, which is expected, since they did not exist.¶
Interoperation. None to report. The mechanism has not been exercised between independent implementations, and the author invites reports from any publisher or consumer willing to attempt it.¶
The following are known to be unresolved and are listed to invite comment rather than to promise resolution.¶
The convention that this document attempts to make discoverable was proposed and popularised by others, and this document takes no credit for it. The measurement of Section 2 was made on the author's own hosts, with the limitations that implies and which Section 2.3 states.¶