Across four independently analysed markets, whole-site exclusion of AI retrieval crawlers remains uncommon, while restriction reaching primary public content stays in the low double digits. Most of what a binary count calls “blocking AI” falls on operational and secondary paths — not on the content AI systems retrieve to answer questions.
A binary count records 34–44% of policy-observed domains as “blocking AI.” Read by the functional purpose of the restricted path, whole-site exclusion is just 3.8–5.0%, and restriction reaching primary public content is 8.0–11.7%.
The binary headline is not false — these directives exist — but it conflates a site restricting its checkout API or admin path with a site withholding its primary public content. Most observed restriction is the former.
The same robots.txt policy data, read at three levels of stringency, from the binary count down to whole-site exclusion. Every figure is drawn from the four frozen country volumes. The staircase has the same shape in every market.
Read down any market and the staircase is the same shape: a binary count in the high-30s to low-40s collapses to single-digit-to-low-double-digit meaningful restriction (8.0–11.7%) and ~4–5% whole-site exclusion. Every measure shown is final. A broader “expanded” measure is held as a documented programme output pending unknown-path review and is not used in these principal findings.
This flagship introduces no new data. Every figure is drawn from an independently frozen country volume. The country volumes are descriptive; comparison and synthesis live only here, in the flagship.
Strict exclusion clusters in a 1.2-point band (3.8–5.0%) and meaningful restriction in a 3.7-point band (8.0–11.7%) across four economies that differ in language convention, data-protection regime, and hosting infrastructure. The gap between the binary headline and the functional reading is not a national peculiarity — it recurs everywhere.
Observed consistency, not tested equivalence. This is a consistency of pattern across four frozen samples, each a property of its own denominator. It is not a claim that the markets are statistically indistinguishable; no formal between-market test is performed. The claim is exactly as strong as the evidence supports — no more.
Claims that businesses are broadly withholding their public content from AI systems are not supported by these four samples. Wholesale exclusion is rare; primary-content restriction is the exception, not the rule. A site can disallow its entire cart and account system and remain fully visible to an AI assistant summarising its products. The binary count cannot tell that situation apart from a site that genuinely withholds its primary content — so it overstates the second by counting the first.
The appropriate correction to the public conversation is not that businesses do not restrict AI crawlers at all, but that the restriction they apply is, in the great majority of cases, directed away from their primary public content rather than at it.
Supersedes the v1.2 binary reading. An earlier version of this study (v1.2) reported a pooled binary rate of ~40% and led with “blocking is common.” That reading is now superseded. The methodological transition is documented in GDR-002; the v1.2 pages remain available as the historical record. View the archived v1.2 cross-market page →
Roles. This study is published by the Periodic Table of Digital Authority (PTODA), the publisher and steward of the PTODA research methodology. It was conducted using the PTODA C01 Crawler v1.5 / v1.5.1, a deterministic robots.txt reference instrument, under PTODA C01 Crawler Methodology v1.5. Samples were constructed from named public sources using the published sampling standard. Commercial relationships played no role in domain selection, inclusion, exclusion, analysis, or interpretation. The methodology is fully documented and designed to support independent reproduction using the published specification and frozen datasets. This study publishes aggregate, anonymised findings only.
Attribution chain: Douglas Lord (researcher and author) · Periodic Table of Digital Authority (publisher and methodology steward) · PTODA C01 Crawler v1.5 (research instrument) · Digital Dominator Pty Ltd ABN 28 616 931 116 (operating entity).
Intellectual property notice: This study, its methodology, findings, data, and all associated content are the original work of Douglas Lord and the property of Digital Dominator Pty Ltd (ABN 28 616 931 116). The Periodic Table of Digital Authority™ is a coined framework and trade mark pending (TM 2644497). AUTHORITY44™ is a trade mark pending (TM 2643932). All rights reserved.