Blocklist · Webz.io

Sites that block omgilibot

2,164 of the 29,991 domains we have measured, 7%, disallow omgilibot at the site root in robots.txt. omgilibot collects content that may be used to train a model.

Every domain here has withheld its content from Webz.io's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

2,164 of 29,991 measured domains (7%) disallow omgilibot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block omgilibot", measured 2026-11-21. https://crawlcensus.com/blocked/omgilibot

The list 2,164 domains

RankDomainTLDAI access score
27,256 amny.com .com 83
27,275 unenvironment.org .org 68
27,306 stylecaster.com .com 77
27,315 fourfourtwo.com .com 84
27,344 ga.de .de 80
27,357 squareblogs.net .net 57
27,361 westfalen-blatt.de .de 81
27,401 derstandard.de .de 56
27,406 warbyparker.com .com 76
27,425 carbuzz.com .com 85
27,467 wfmz.com .com 81
27,482 tug.org .org 44
27,551 latostadora.com .com 65
27,608 cortexbid.com .com 72
27,660 pokelike.xyz .xyz 65
27,696 staradvertiser.com .com 84
27,723 geek.com .com 38
27,756 beatstars.net .net 51
27,796 asiasociety.org .org 67
27,833 apotheken-umschau.de .de 45
27,838 mtvuutiset.fi .fi 72
27,891 blogto.com .com 67
27,906 usip.org .org 70
27,946 trustedshops.de .de 73
27,999 earth.com .com 80
28,003 footballant.com .com 75
28,022 direct-book.com .com 45
28,038 topspeed.com .com 81
28,047 biodiversitylibrary.org .org 50
28,074 wvdme.com .com 49
28,079 apsense.com .com 52
28,099 explained.today .today 42
28,170 suedkurier.de .de 81
28,177 gazette.com .com 80
28,220 icecat.se .se 52
28,290 icecat.dk .dk 56
28,320 laptopmag.com .com 87
28,336 cargoagent.net .net 51
28,349 suamusica.com.br .br 77
28,361 core77.com .com 57
28,377 menuweb.menu .menu 69
28,383 lightreading.com .com 81
28,438 360cities.net .net 49
28,488 jsr.io .io 61
28,522 distance.to .to 57
28,573 wpr.org .org 88
28,582 charentelibre.fr .fr 82
28,616 amzn.com .com 40
28,642 nachrichten.at .at 82
28,676 xforum.live .live 60
28,678 thenational.scot .scot 81
28,705 wellcomecollection.org .org 45
28,729 famelack.com .com 81
28,740 ctt.pt .pt 54
28,754 nekobt.to .to 54
28,757 federalnewsnetwork.com .com 86
28,792 uesp.net .net 55
28,794 eastbaytimes.com .com 79
28,868 myvolumio.org .org 38
28,895 weatheronline.co.uk .uk 47
28,935 red-dot.org .org 74
28,996 sebokep.vip .vip 78
29,011 fraserinstitute.org .org 51
29,031 threadreaderapp.com .com 59
29,063 golzu.com .com 54
29,115 gp.se .se 78
29,121 pucsp.br .br 55
29,139 u-cursos.cl .cl 51
29,166 freakonomics.com .com 74
29,170 spinmagazine.com .com 79
29,173 tudosaladeaula.com .com 91
29,260 b-cd.app .app 52
29,273 expressnews.com .com 54
29,293 ceu.edu .edu 58
29,313 qbittorrent.org .org 50
29,323 goop.com .com 66
29,340 bunjang.co.kr .kr 62
29,341 geometrylitepc.io .io 70
29,390 kompas.id .id 64
29,402 mmafighting.com .com 68
29,447 artofproblemsolving.com .com 86
29,499 conjur.com.br .br 73
29,513 mindtos.com .com 57
29,551 lavoro.gov.it .it 63
29,553 hankookilbo.com .com 84
29,562 therarbg.com .com 77
29,567 ctpost.com .com 54
29,572 flashscore.com.mx .mx 70
29,634 westword.com .com 77
29,639 edmontonjournal.com .com 71
29,661 nn125.com .com 49
29,692 piliapp.com .com 65
29,700 gamessofun.com .com 50
29,821 providencejournal.com .com 73
29,884 techtudo.com.br .br 84
29,980 lidovky.cz .cz 67
29,991 efhub.com .com 55
30,004 sfstandard.com .com 72
30,049 processing.org .org 57
30,074 yugipedia.com .com 58
30,161 bostonmagazine.com .com 72
30,202 serializd.com .com 65
30,237 clarionledger.com .com 74
30,288 anycubic.com .com 68
30,298 afnor.org .org 81
30,301 qtoon.org .org 61
30,305 telnyx.com .com 94
30,356 edaily.co.kr .kr 74
30,376 getxray.app .app 80
30,382 wook.pt .pt 57
30,387 bsc.es .es 37
30,393 ecode360.com .com 44
30,399 spoton.com .com 94
30,443 dobrzemieszkaj.pl .pl 61
30,470 turbo.cr .cr 54
30,500 routard.com .com 78
30,507 abc11.com .com 81
30,517 premioibest.com .com 70
30,588 streamimdb.ru .ru 72
30,619 spectrumnews1.com .com 45
30,678 xbato.co.uk .uk 73
30,681 siasat.com .com 84
30,698 nswpedia.com .com 75
30,703 wickedlocal.com .com 75
30,711 hi5.com .com 43
30,712 artforum.com .com 74
30,738 fedpat.com.ar .ar 69
30,828 bowdoin.edu .edu 71
30,864 umms.org .org 76
30,915 sendtuna.com .com 45
30,928 bokepindoh.party .party 71
30,976 tablegame.ph .ph 39
30,986 commercialappeal.com .com 74
30,992 huffingtonpost.it .it 83
31,003 maxim.com .com 87
31,023 phoenixnewtimes.com .com 71
31,032 howlongtobeat.com .com 49
31,057 linksbird.co .co 54
31,145 knowing-jesus.com .com 70
31,149 idealhome.co.uk .uk 82
31,176 edp24.co.uk .uk 79
31,195 birdlife.org .org 73
31,231 grupofpsinais.bet .bet 54
31,232 omicaso.org .org 58
31,282 vndb.org .org 60
31,284 publicdomainpictures.net .net 68
31,293 mcsweeneys.net .net 78
31,320 lumendatabase.org .org 51
31,373 yugiyu2.com .com 56
31,391 xn--l3cly6b0fd0c.net .net 76
31,392 batikloreng.online .online 76
31,425 ewtnnews.com .com 70
31,434 xxxtubevideos.click .click 61
31,441 yacenetv.com .com 83
31,443 saarbruecker-zeitung.de .de 82
31,445 kingviral.baby .baby 72
31,458 cricfytvap.net .net 78
31,475 tmll7.com .com 53
31,499 bolasport.com .com 76
31,533 krishitalk.com .com 56
31,549 lohud.com .com 74
31,565 alexiskian.com .com 70
31,566 apinagadigital.com .com 64
31,571 dailyroutine.lifestyle .lifestyle 30
31,582 gamenila.com .com 56
31,591 lottosociety.com .com 75
31,593 me.game .game 36
31,598 pahindot.me .me 77
31,602 quickplayhub.in .in 56
31,623 videyli.co .co 26
31,628 xvidey.top .top 53
31,638 marieclaire.co.uk .uk 84
31,686 one-name.net .net 29
31,758 are.na .na 72
31,782 mastodon.online .online 49
31,786 thespec.com .com 64
31,796 winespectator.com .com 82
31,799 wefut.com .com 50
31,850 avsforum.com .com 43
31,871 diariodesevilla.es .es 84
31,908 lamag.com .com 78
31,986 gamemaker.io .io 72
32,000 sciencesetavenir.fr .fr 71
32,032 prx.org .org 74
32,125 elmercurio.com .com 43
32,169 journaldugeek.com .com 91
32,183 tellmyroute.com .com 72
32,263 shacknews.com .com 73
32,267 theargus.co.uk .uk 79
32,280 zaobao.com .com 87
32,345 opencv.org .org 66
32,388 rgj.com .com 73
32,391 prweek.com .com 51
32,437 hoopshype.com .com 70
32,456 currencyrate.today .today 67
32,460 saturn.de .de 91
32,484 project1999.com .com 47
32,498 toptenreviews.com .com 87
32,579 dallasobserver.com .com 77
32,595 saveur.com .com 70
PreviousPage 7 of 11Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the omgilibot reference, the full census, and the JSON endpoint.