How many AI tools block GPTBot?

In our directory sample, 10.5% of tools explicitly disallowed GPTBot in robots.txt. Named blocks for the ChatGPT search crawler were much less common. This measures the published rules of tools listed on directree, not all AI products or actual crawler access.

We crawled the robots.txt of every tool listed on directree on 6 and 7 September 2026. Of the 9,037 that answered, 945 name GPTBot in their own rule and disallow it entirely. That is 10.5%. For Anthropic's ClaudeBot it is 10.1%, for Google-Extended 9.9%. ByteDance's Bytespider is refused by 11.8% of sites, and Common Crawl's CCBot by 11.3%.

Named rules differ between training and search crawlers

Crawler	What it feeds	Sites that block it
Bytespider	ByteDance training	11.8%
CCBot	Common Crawl datasets	11.3%
GPTBot	OpenAI training	10.5%
Amazonbot	Amazon	10.5%
meta-externalagent	Meta training	10.4%
ClaudeBot	Anthropic training	10.1%
Google-Extended	Gemini training	9.9%
Applebot-Extended	Apple training	9.9%
ChatGPT-User	ChatGPT browsing on behalf of a user	0.3%
PerplexityBot	Perplexity search	0.3%
OAI-SearchBot	ChatGPT search results	0.2%
Claude-SearchBot	Claude search results	0.1%

Of the 945 sites that block GPTBot, 839 (88.8%) still allow OAI-SearchBot, and 903 (95.6%) also block ClaudeBot. These are observed robots.txt configurations. They do not establish the owners' intentions or whether a crawler could fetch a page through other access controls.

Only 31 sites (0.3%) block all crawlers outright. A further 44% have some path-level Disallow rule in their wildcard group, usually for an API or admin path, which is ordinary hygiene and not an AI policy. We count those separately so the 10.5% counts named crawler rules. A rule may be written by an owner, a template or a hosting service.

Block rates by hosting provider

Hosting	Sites	Block GPTBot
Cloudflare	3,749	22.4%
Vercel	2,556	5.2%
Netlify	273	4.0%
Fastly	361	3.9%
CloudFront	328	3.4%

The observed rate for Cloudflare-fronted sites is about 4.3 times the Vercel rate in this sample. This is a correlation. We did not inspect account settings or control for category, domain age or other differences between the groups.

Cloudflare offers both managed robots.txt and separate request-blocking controls. Its Block AI Bots documentation describes defaults updated on September 15, 2026 for new domains: training and agent bots are blocked on pages with ads, while search bots are allowed. Our September 6 and 7 crawl predates that update and cannot measure its effect. It also cannot tell us which rules an owner chose manually.

Younger sites block more

Domain registered	Sites	Block GPTBot
2026	2,900	13.6%
2025	836	10.2%
2024	229	7.9%
2023	145	6.9%

Sites registered this year block GPTBot twice as often as sites from 2023. These groups have different sample sizes, and we have not separated domain age from hosting or category.

Creative tools block most, marketing tools least

Among categories with at least 150 crawled tools, design and media tools sit at the top and marketing tools at the bottom.

Category	Sites	Block GPTBot
Design and UI	318	17.3%
Video and audio	330	16.1%
AI image and video generation	426	14.3%
Community and membership	237	14.3%
Productivity	695	11.9%
Developer tools	273	9.9%
Automation	210	8.1%
Education and e-learning	389	8.0%
Marketing and SEO	245	5.3%

We did not survey owners or test why they use these rules.

What this means if you run an AI tool

OpenAI documents GPTBot and OAI-SearchBot as independently configurable crawlers. A site can disallow the training crawler while allowing the search crawler. Allowing a crawler does not guarantee a citation or traffic. ChatGPT-User handles user-initiated actions and is a separate mechanism.

Check your published robots.txt and any hosting-level bot controls together. This crawl records robots.txt rules; it does not test enforcement, past training use, indexing or appearance in AI answers.

Method and limits

The crawl covered directree's published listings on September 6 and 7, 2026: 9,160 tools, with usable responses for 9,037. The 123 without a usable response are excluded. This is a directory sample, not a random sample of all websites or AI tools. The headline counts a named crawler group with Disallow: /; blanket wildcard blocks and partial-path rules are separate measures. Missing named blocks do not by themselves prove access is allowed.

Hosting labels come from headers and DNS. Domain registration dates come from RDAP. The breakdowns describe associations; no causal effect or owner intent was measured. The article's numbers remain the September 7 snapshot, even if the live directory grows.

Editorial update, September 15, 2026: clarified the sample and crawler-access limits, corrected the description of Cloudflare defaults, and removed unsupported explanations of owners' choices. The crawl figures have not changed.

Disclosure: directree operates the directory used for this study. Browse the directory to see the product listings behind this work.