Every other number on this site reads what a website asks crawlers to do. This one reports what happened next, for the only site where both halves are visible: this one. The rules are three lines of our own robots.txt — /scan /api/v1/scan /claim — and every request reaching a path they forbid is counted here.

It exists because a claim was made here without being checked. /claim was disallowed to stop this site generating the wasted requests it publishes lists about elsewhere, and the change was reported as finished. It was not measured. Some time later the day's total on that path stood at 922 requests from AI crawlers, which looked like the rule being ignored.

It probably was not. Over the following forty minutes that counter moved from 922 to 922 — not one further request. Crawlers cache robots.txt, commonly for a day, so a client holding a copy fetched before the rule existed will keep requesting the path until it refreshes. A burst that stops is what compliance looks like from the origin; a rate that continues is what defiance looks like. Reading a cumulative total as either one was the same error twice: asserting an outcome without an instrument. The instrument is what this page is.

These requests came from clients Cloudflare verified as the crawler they said they were, against published address ranges and reverse DNS rather than the user-agent header. This is the only table on the page that says anything about a named company.

Verified clients on disallowed paths: none recorded.

These carried a crawler's user-agent string and were not verified as that crawler. They are published because the volume matters to anyone reading a robots.txt compliance figure anywhere, and withheld from attribution because the header is worthless as evidence. The first two rows this table ever recorded claimed to be GPTBot and ClaudeBot and arrived from a residential ISP: they were test requests from the laptop that built this page.

If your crawler appears here and should not, the address ranges it uses are probably not published where Cloudflare can verify them, and that is worth fixing for reasons well beyond this page.

Requests reaching the same paths without any crawler's user-agent token. They are counted because hiding them would overstate how clean the picture is, and kept out of the figures above because they are not crawlers ignoring a rule. The largest contributor is this project's own test suite, which calls /api/v1/scan on purpose: 36 of the first 37 rows this table ever recorded. Filtering our own address out instead of reporting it would be the quiet exception this census exists to catch other people making.

• A single request proves nothing, and neither does a burst. A crawler holding a URL discovered before a rule existed, or a cached copy of the file that predates it, will keep fetching for as long as its cache lasts. Only a rate that persists past that is evidence of anything.
• This is one site, with three rules, and no claim is made that it generalises. It is offered because first-party observation of robots.txt enforcement is rare, not because the sample is good.
• The unverified column names no company. Anyone can send any header.
• The data is available as JSON under CC BY 4.0, with the same figures.