How many AI tools block GPTBot?

One in ten AI tools tells OpenAI's training crawler to go away. Almost none of them tell the ChatGPT search crawler the same thing.

We crawled the robots.txt of every tool listed on directree on 6 and 7 September 2026. Of the 9,037 that answered, 945 name GPTBot in their own rule and disallow it entirely. That is 10.5%. For Anthropic's ClaudeBot it is 10.1%, for Google-Extended 9.9%. The most blocked crawler is not from an AI lab at all: ByteDance's Bytespider is refused by 11.8% of sites, and Common Crawl's CCBot by 11.3%.

The training bots get blocked, the search bots do not

Crawler	What it feeds	Sites that block it
Bytespider	ByteDance training	11.8%
CCBot	Common Crawl datasets	11.3%
GPTBot	OpenAI training	10.5%
Amazonbot	Amazon	10.5%
meta-externalagent	Meta training	10.4%
ClaudeBot	Anthropic training	10.1%
Google-Extended	Gemini training	9.9%
Applebot-Extended	Apple training	9.9%
ChatGPT-User	ChatGPT browsing on behalf of a user	0.3%
PerplexityBot	Perplexity search	0.3%
OAI-SearchBot	ChatGPT search results	0.2%
Claude-SearchBot	Claude search results	0.1%

Of the 945 sites that block GPTBot, 839 (88.8%) still allow OAI-SearchBot, and 903 (95.6%) also block ClaudeBot. The policy is the same for every lab: no training on our pages, but do send us traffic.

Only 31 sites (0.3%) block all crawlers outright. A further 44% have some path-level Disallow rule in their wildcard group, usually for an API or admin path, which is ordinary hygiene and not an AI policy. We count those separately so the 10.5% is an explicit, named decision.

Cloudflare hosting is the biggest single factor

Hosting	Sites	Block GPTBot
Cloudflare	3,749	22.4%
Vercel	2,556	5.2%
Netlify	273	4.0%
Fastly	361	3.9%
CloudFront	328	3.4%

A tool on Cloudflare is more than four times as likely to block GPTBot as a tool on Vercel. The likely reason is not that Cloudflare customers hold different views: Cloudflare offers a one-click "block AI bots" setting and a managed robots.txt that writes these rules, and since 15 September 2026 it blocks AI crawlers by default for new sites. Most of the policy in this dataset was set by a toggle, not by a decision.

The 10.5% is measured before Cloudflare's default flipped. We will recompute this page on the same query after the change lands across the crawl and report the difference.

Younger sites block more

Domain registered	Sites	Block GPTBot
2026	2,900	13.6%
2025	836	10.2%
2024	229	7.9%
2023	145	6.9%

Sites registered this year block GPTBot twice as often as sites from 2023. The newest sites are disproportionately on Cloudflare-fronted stacks with the toggle on, and blocking AI crawlers became a default in template robots.txt files during 2025.

Creative tools block most, marketing tools least

Among categories with at least 150 crawled tools, design and media tools sit at the top and marketing tools at the bottom.

Category	Sites	Block GPTBot
Design and UI	318	17.3%
Video and audio	330	16.1%
AI image and video generation	426	14.3%
Community and membership	237	14.3%
Productivity	695	11.9%
Developer tools	273	9.9%
Automation	210	8.1%
Education and e-learning	389	8.0%
Marketing and SEO	245	5.3%

Design, video and image tools produce the kind of content models train on, and they opt out most. Marketing tools depend on being found, and they block least.

What this means if you run an AI tool

Blocking GPTBot does not remove you from ChatGPT's answers. Those come through OAI-SearchBot and ChatGPT-User, which use separate rules and which almost nobody blocks. The configuration most blockers already run, GPTBot disallowed and OAI-SearchBot allowed, keeps pages out of training data and keeps them citable. Check that a hosting toggle has not written your policy for you: on Cloudflare, more than one in five sites in this dataset block the training crawler, and many of their owners will not know.

Your tool is in this dataset if it is listed on directree. Claiming the listing lets you confirm its facts and adds it to the founder-confirmed cuts.