# llms.txt on 400 websites: data and reproduction

Report: https://www.directree.io/research/llms-txt-400-websites

Published 23 September 2026 (Stockholm). Captures: 22 September 2026, 22:58–23:01 UTC.

## Files

- `audit.csv`: 400 sampled hosts, request outcomes, timestamps, SHA-256 hashes, structural checks and editorial classifications.
- `sampling-frame.csv`: all 11,457 eligible normalized hostnames, frozen before collection.
- `summary.json`: sample seed, collection window and counts.
- `protocol.md`: collection and review rules, including the review-stage clarification.
- `reproduce.py`: offline verification of the sample and headline arithmetic.
- `sampling-frame.sql`: read-only query used to retrieve the population from Directree's private database.
- `headline.sql`: read-only query computing the headline from the stored, frozen observations. Database access is private; the CSV permits the same count without access.

Download the first three files and `reproduce.py` into one folder, then run:

```sh
python3 reproduce.py
```

Expected output includes `hits: 181`, `n: 400`, `pct: 45.3` and `sample_verified: true`. Only Python's standard library is required.

The script verifies the first 400 hosts in ascending SHA-256 order using the seed in `summary.json`, with no replacement or selection by previous llms.txt flags. It recomputes the published counts from the audit labels. It does not independently verify those labels or re-fetch today's websites.

## Definitions

`h1_text_candidate` means HTTP 200, non-HTML, nonempty text whose first non-whitespace content, after an optional UTF-8 BOM, is a Markdown H1. It is not a complete specification-validity verdict. Other HTTP responses, missing root files, HTML, network failures and empty/binary responses have separate labels.

`primary_positive` additionally requires project information in the file and matching project content in the separately fetched homepage body. A page title alone is insufficient. “Project” includes services, stores, publications and prelaunch products. Neither operation nor individual marketing claims were tested.

The Codex research assistant reviewed the 203 successful text candidates. `file_purpose`, `homepage_review` and `review_note` are AI-assisted editorial labels, without independent human review. The other 197 homepages were captured but not editorially classified. `not_reviewed_non_candidate` means that no homepage or file-purpose review was required after the endpoint screen failed.

`unknown` means the homepage could not be assessed from the capture; it does not mean the product is inactive. `possibly_truncated` flags captures that reached the 512 KiB limit. A null HTTP status means no HTTP response was recorded. Public URLs omit query strings and fragments; the private ledger retains exact requested and final URLs.

The structural feature checks use Markdown patterns. In particular, bare URLs do not count as Markdown links. Neither outgoing links nor full proposal compliance were tested. All 400 hosts stay in the denominator. Percentages use conventional half-up rounding to one decimal place.

Stored flags describe a different observation at another time. Their disagreement with this stricter audit is not evidence of adoption growth or decline. Recomputing this frozen study does not refresh its underlying observations.

## Evidence and reuse

Raw homepage and file bodies are stored privately with their timestamps and hashes. The public dataset does not redistribute third-party page text. Raw captures are necessary to independently re-adjudicate the historical labels; the public package supports sample and arithmetic reproduction.

Directree's audit data, labels and original report are available under CC BY 4.0, consistent with the research page's dataset metadata. Third-party site content and trademarks retain their respective rights.

Cite: Directree Research, “llms.txt on 400 websites: what a fresh check found,” published 23 September 2026, observations 22 September 2026 UTC. https://www.directree.io/research/llms-txt-400-websites