llms.txt on 400 websites: what a fresh check found

Of 400 Directree-listed websites checked in a fresh crawl, 181 served an llms.txt document describing the project, service or business shown on their homepage. That is 45.3% of the sample.

To count toward that result, the file had to return HTTP 200, contain non-HTML text and start with a Markdown H1 heading. We also checked its subject against the separately fetched homepage body. A file passing this screen is not necessarily correct, current or compliant with every part of the llms.txt proposal.

The extra content check mattered. Five files with the same basic structure described domains for sale. Four others described projects whose homepage content we could not assess. Both groups remain in the 400-site denominator but sit outside the headline count.

Download the 400-row audit, or reproduce the sample and counts.

What came back from /llms.txt

We requested /llms.txt at the origin of each selected listing URL, following up to five redirects. The table groups every selected host once.

Percentages are rounded to one decimal place, so the individual rounded values do not add to exactly 100%.

A 404 or 410 applies to the root URL we requested. It does not show that a project has no file on a documentation subdomain or under another path. A failed request is also different from an HTTP response saying the file is missing.

A well-formed-looking file can describe a domain sale

Five of the 190 files that passed the HTTP, text and initial-heading checks explicitly described a domain being for sale. Counting all 190 as software adoption would include those sale documents.

Their homepage responses had no readable body content. We therefore label the file as a domain-sale document and leave the homepage unassessed. We do not use the file alone to declare a former product closed.

After removing those five, 185 H1-first files described a project, service or business. Two corresponding homepages returned browser challenges. Two more supplied no readable homepage body within our capture, including one that reached the response-size limit. That leaves the 181 matching cases in the headline.

This is deliberately a narrower count than file availability. A business can have a useful document even when our crawler cannot read its homepage. The audit preserves those cases so readers can apply a different rule without repeating the crawl.

Six more files described projects without starting with an H1

Thirteen successful text responses did not begin with a Markdown H1. Six contained project or website information; seven contained crawler-style directives rather than a project description.

Some of the six began with generator text before a later heading. Others used a plain-text title. We record them separately instead of treating every text response as equivalent or silently discarding them as failed requests.

The llms.txt proposal describes a Markdown document beginning with an H1. It also permits files under subpaths, which this root-path survey did not search. Our initial-heading test is a limited structural screen, not a complete implementation of the proposal.

Among the 181 matching files, 163 contained a blockquote, 177 contained an H2 section and 148 contained at least one Markdown-formatted link. These are descriptive checks; none was an additional requirement for our headline. Plain URLs do not count as Markdown-formatted links in this breakdown, and linked destinations were not tested.

How we chose and reviewed the sample

We froze the population at 22:57 UTC on 22 September 2026: 11,457 published Directree listings with a hostname and an HTTP(S) URL, representing 11,457 distinct normalized hosts. We lowercased hostnames and removed a leading www. label.

We then hashed a fixed seed plus each hostname and selected the first 400 in hash order. Selection did not depend on the existing llms.txt flag, an earlier liveness result or whether the request succeeded. We made no replacements.

The crawl ran from 22:58 to 23:01 UTC on 22 September, which was already 23 September in Stockholm. That is why the report's publication date and UTC data date differ.

Each homepage and root file received one initial request, with a 15-second timeout, up to five redirects and a 512 KiB decoded-body cap. We did not log in, render JavaScript or bypass browser challenges. Twenty-one homepage captures reached the cap; no llms.txt capture did.

The research assistant reviewed all 203 successful text candidates against their captured homepage evidence. File-purpose and homepage-match labels are AI-assisted editorial judgments, without an independent human audit. HTTP statuses, timestamps, sizes and hashes are recorded observations. Neither type of evidence establishes that a product works or that its marketing claims are true. Prelaunch sites and service businesses can qualify when their file and homepage describe the same project.

The protocol records the precollection rules and a review-stage clarification: a page title alone did not establish a homepage match. Raw response bodies are retained privately; the public audit contains classifications, timestamps and body hashes rather than republished page text.

What this says about llms.txt adoption

It describes this sample of Directree-listed websites at one point in time. The directory includes software, directories, services, stores and other projects. The sample is not representative of every website or of AI products alone, so 45.3% should not be quoted as global llms.txt adoption.

Stored presence flags and fresh classifications also answer different questions. Within the sample, 202 hosts had a stored positive flag. Of those, 178 met our stricter headline rule. Three stored-negative hosts also met it. Differences can reflect timing, failed requests, response content and the new rubric; they do not measure adoption growth or decline.

We did not measure crawler visits, AI citations, search rankings or traffic. This survey cannot show whether publishing an llms.txt file affects any of them.

For researchers measuring availability, the useful distinction is between a response, a document and a document about the project being studied. Keeping those stages separate made five sale documents and four unassessable homepages visible in this result.

Data and reproduction

• Audit CSV: all 400 selected hosts, including unsuccessful requests and review notes.
• Frozen sampling frame: all 11,457 eligible normalized hostnames.
• Protocol and review clarification, with collection limits and label definitions.
• Reproduction instructions, counting script, summary JSON and stored headline SQL.

The headline is recomputed from 400 stored observations tied to this frozen study. A refresh requires a new capture and content review; rerunning the counting query alone does not refresh the evidence.

For other Directree datasets, see our website framework study and September 7 GPTBot blocking snapshot.