Crawl Census
Scan a site
Publishing platform segment

AI crawler access on wordpress sites

490 of the 1,907 sites measured in this segment refuse a request from an identified AI user agent at the edge, 26% of the segment, against 5% whose robots.txt disallows an answer engine at the site root. The block that costs these sites citations is not the one written in the file. That answer-engine rate is 1.2 points below the corpus rate of 6%, which is itself measured across 29,465 domains.

Mean AI access score across those 1,907 sites is 80.8 of 100, 306 of them publish an llms.txt (16%), and 1,572 publish structured data an answer engine can parse (82%).

1,907
sites measured
80.8
mean AI access score of 1,907
5%
block an answer engine (101 of 1,907)
16%
publish llms.txt (306 of 1,907)

Block rate by crawler, inside this segment robots.txt at the site root

CrawlerOperatorContent used forBlock rateBlockingMeasured
GPTBot OpenAI training 15.3% 292 1,907
CCBot Common Crawl Foundation archive 14.7% 281 1,907
Bytespider ByteDance training 14.5% 276 1,907
ClaudeBot Anthropic training 14.3% 273 1,907
Google-Extended Google training 13.7% 262 1,907
Applebot-Extended Apple training 13.4% 256 1,907
Meta-ExternalAgent Meta training 13.3% 254 1,907
Amazonbot Amazon training 13.2% 251 1,907
anthropic-ai Anthropic training 4.6% 88 1,907
FacebookBot Meta training 4.1% 79 1,907
omgilibot Webz.io training 4.1% 78 1,907
omgili Webz.io training 4.0% 77 1,907
PerplexityBot Perplexity answer index 4.0% 76 1,907
ChatGPT-User OpenAI live retrieval 4.0% 76 1,907
Diffbot Diffbot training 3.8% 73 1,907
Timpibot Timpi training 2.8% 53 1,907
OAI-SearchBot OpenAI answer index 2.7% 51 1,907
YouBot You.com answer index 2.5% 47 1,907
cohere-ai Cohere live retrieval 2.4% 45 1,907
AI2Bot Allen Institute for AI training 1.7% 33 1,907
SemrushBot-OCOB Semrush training 1.7% 32 1,907
PanguBot Huawei training 1.7% 32 1,907
Meta-ExternalFetcher Meta live retrieval 1.6% 31 1,907
cohere-training-data-crawler Cohere training 1.6% 31 1,907
Claude-User Anthropic live retrieval 1.6% 31 1,907
Scrapy Zyte (open-source framework) training 1.6% 30 1,907
Kangaroo Bot Kangaroo LLM training 1.6% 30 1,907
Perplexity-User Perplexity live retrieval 1.5% 29 1,907
Claude-SearchBot Anthropic answer index 1.5% 28 1,907
DuckAssistBot DuckDuckGo live retrieval 1.4% 27 1,907
MistralAI-User Mistral AI live retrieval 1.3% 25 1,907
Applebot Apple answer index 1.3% 25 1,907
Amzn-SearchBot Amazon answer index 1.3% 25 1,907
ImagesiftBot ImageSift (Hive) training 0.9% 18 1,907
Google-CloudVertexBot Google live retrieval 0.5% 10 1,907
GoogleOther Google training 0.5% 9 1,907
Amzn-User Amazon live retrieval 0.3% 6 1,907
TikTokSpider ByteDance live retrieval 0.3% 5 1,907
Meta-WebIndexer Meta answer index 0.3% 5 1,907
MistralAI-Index Mistral AI answer index 0.1% 1 1,907
Googlebot-News Google answer index 0.1% 1 1,907
Googlebot Google answer index 0.1% 1 1,907
OAI-AdsBot OpenAI live retrieval 0.0% 0 1,907
msnbot Microsoft answer index 0.0% 0 1,907
MistralAI-Training Mistral AI training 0.0% 0 1,907
Meta-ExternalAds Meta training 0.0% 0 1,907
facebookexternalhit Meta live retrieval 0.0% 0 1,907
Bingbot Microsoft answer index 0.0% 0 1,907

Block rate is the share of this segment's domains holding a policy record for that agent whose robots.txt disallows it at the site root, shown with its numerator and denominator. Denominators differ between agents because a record only exists once a domain has been measured for it. A domain serving no robots.txt counts as allowing everything, which is what the standard specifies.

Most accessible

DomainScoreEngines blockedllms.txt
1 dataimpulse.com 99 0 yes
2 textexpander.com 99 0 yes
3 thesouthafrican.com 98 0 yes
4 webkul.com 98 0 yes
5 acculynx.com 98 0 yes
6 infoblox.com 98 0 yes
7 civicplus.com 98 0 yes
8 19fortyfive.com 98 0 yes
9 javgg.net 97 0 yes
10 transcy.io 97 0 yes
11 resonate.com 97 0 yes
12 tenjin.io 97 0 yes
13 chainstack.com 97 0 yes
14 sumologic.com 97 0 yes
15 hostinet.com 97 0 yes

Highest AI access scores among the 1,907 sites measured in this segment.

Least accessible

DomainScoreEngines blockedllms.txt
1 kyalsin.com 41 0 no
2 nationalreview.com 44 0 no
3 skidrowreloaded.com 45 0 no
4 startersites.io 46 5 no
5 cmpunjabrashancard.pk 46 0 no
6 cocomic.co 47 0 no
7 javmyanmar.com 47 0 no
8 novelupdates.com 48 0 no
9 petronasnearme.my 48 0 no
10 2trk.info 48 0 no
11 makip.co.jp 48 0 no
12 bfd269.com 49 0 no
13 primefaces.org 49 0 no
14 pinayflix.app 49 0 no
15 pinayflix.uk 49 0 no

Lowest AI access scores among the same 1,907 sites.

Compare with the rest of the grouping publishing platform segments

next.js 1,612 sites · 7% blockdrupal 842 sites · 4% blockclient-rendered shell 604 sites · 3% blocknuxt 508 sites · 8% blockgatsby 137 sites · 1% blockastro 119 sites · 3% blockhugo 102 sites · 1.0% blockwebflow 90 sites · 8% blockjoomla 45 sites · 7% blockwix 34 sites · 0% block

The largest 10 other segments in this grouping, ordered by size. The share beside each is that segment's own answer-engine block rate over its own count. Full table on the segments index.

What this segment does not tell you read this before citing it

Platform detection relies on a generator meta tag, so coverage is partial and skewed. A site that strips the tag, or a system that never writes one, is invisible here. These rows therefore over-represent platforms that advertise themselves and say nothing at all about the ones that do not.

Measure your own domain

The audit that produced every number on this page runs on any domain in about six seconds: robots.txt resolved for 48 agents, then live requests sent as four of them to see whether the edge agrees with the file.